Petuum: A New Platform for Distributed Machine Learning on Big Data

Eric P. Xing1, Qirong Ho2, Wei Dai1, Jin Kyu Kim1, Jinliang Wei1, Seunghak Lee1, Xun Zheng1,Pengtao Xie1, Abhimanu Kumar1, and Yaoliang Yu1

1School of Computer Science, Carnegie Mellon University2Institute for Infocomm Research, A*STAR, Singapore

{epxing,wdai,jinkyuk,jinlianw,seunghak,xunzheng,pengtaox,yaoliang}@cs.cmu.edu{hoqirong,abhimanyu.kumar}@gmail.com

May 18, 2015

Abstract

What is a systematic way to efficiently apply a widespectrum of advanced ML programs to industrialscale problems, using Big Models (up to 100s of bil-lions of parameters) on Big Data (up to terabytes orpetabytes)? Modern parallelization strategies employfine-grained operations and scheduling beyond the classicbulk-synchronous processing paradigm popularized byMapReduce, or even specialized graph-based executionthat relies on graph representations of ML programs.The variety of approaches tends to pull systems andalgorithms design in different directions, and it remainsdifficult to find a universal platform applicable to awide range of ML programs at scale. We propose ageneral-purpose framework that systematically addressesdata- and model-parallel challenges in large-scale ML,by observing that many ML programs are fundamen-tally optimization-centric and admit error-tolerant,iterative-convergent algorithmic solutions. This presentsunique opportunities for an integrative system design,such as bounded-error network synchronization anddynamic scheduling based on ML program structure. Wedemonstrate the efficacy of these system designs versuswell-known implementations of modern ML algorithms,allowing ML programs to run in much less time and atconsiderably larger model sizes, even on modestly-sizedcompute clusters.

1 Introduction

Machine learning (ML) is becoming a primary mechanismfor extracting information from data. However, the surg-ing volume of Big Data from Internet activities and sen-sory advancements, and the increasing needs for Big Mod-els for ultra high-dimensional problems have put tremen-dous pressure on ML methods to scale beyond a single

machine, due to both space and time bottlenecks. Forexample, the Clueweb 2012 web crawl1 contains over 700million web pages as 27TB of text data, while photo-sharing sites such as Flickr, Instagram and Facebook areanecdotally known to possess 10s of billions of images,again taking up TBs of storage. It is highly inefficient,if possible, to use such big data sequentially in a batchor scholastic fashion in a typical iterative ML algorithm.On the other hand, state-of-the-art image recognition sys-tems have now embraced large-scale deep learning modelswith billions of parameters [14]; topic models with up to106 topics can cover long-tail semantic word sets for sub-stantially improved online advertising [23, 28]; and very-high-rank matrix factorization yields improved predictionon collaborative filtering problems [32]. Training such bigmodels with a single machine can be prohibitively slow,if possible.

Despite the recent rapid development of many newML models and algorithms aiming at scalable applica-tion [6, 25, 11, 33, 1, 2], adoption of these technologiesremains generally unseen in the wider data mining, NLP,vision, and other application communities for big prob-lems, especially those built on advanced probabilistic oroptimization programs. We suggest that, from the scal-able execution point of view, what prevents many state-of-the-art ML models and algorithms from being morewidely applied at Big-Learning scales is the difficult mi-gration from an academic implementation, often special-ized for a small, well-controlled computer platform suchas desktop PCs and small lab-clusters, to a big, less pre-dictable platform such as a corporate cluster or the cloud,where correct execution of the original programs requirecareful control and mastery of low-level details of the dis-tributed environment and resources through highly non-trivial distributed programming.

1http://www.lemurproject.org/clueweb12.php/

1

arX

iv:1

312.

7651

v2 [

stat

.ML

] 1

4 M

ay 2

015

Figure 1: The scale of Big ML efforts in recent literature. Akey goal of Petuum is to enable larger ML models to be runon fewer resources, even relative to highly-specialized imple-mentations.

Many platforms have provided partial solutions tobridge this research-to-production gap: while Hadoop [24]is a popular and easy to program platform, the simplic-ity of its MapReduce abstraction makes it difficult to ex-ploit ML properties such as error tolerance (at least, notwithout considerable engineering effort to bypass MapRe-duce limitations), and its performance on many ML pro-grams has been surpassed by alternatives [29, 17]. Onesuch alternative is Spark [29], which generalizes MapRe-duce and scales well on data while offering an acces-sible programming interface; yet, Spark does not offerfine-grained scheduling of computation and communica-tion, which has been shown to be hugely advantageous,if not outright necessary, for fast and correct executionof advanced ML algorithms [4]. Graph-centric platformssuch as GraphLab [17] and Pregel [18] efficiently partitiongraph-based models with built-in scheduling and consis-tency mechanisms; but ML programs such as topic mod-eling and regression either do not admit obvious graphrepresentations, or a graph representation may not bethe most efficient choice; moreover, due to limited the-oretical work, it is unclear whether asynchronous graph-based consistency models and scheduling will always yieldcorrect execution of such ML programs. Other systemsprovide low-level programming interfaces [20, 16], that,while powerful and versatile, do not yet offer higher-levelgeneral-purpose building blocks such as scheduling, modelpartitioning strategies, and managed communication thatare key to simplifying the adoption of a wide range of MLmethods. In summary, existing systems supporting dis-tributed ML each manifest a unique tradeoff on efficiency,correctness, programmability, and generality.

In this paper, we explore the problem of building adistributed machine learning framework with a new an-

gle toward the efficiency, correctness, programmability,and generality tradeoff. We observe that, a hallmark ofmost (if not all) ML programs is that they are defined byan explicit objective function over data (e.g., likelihood,error-loss, graph cut), and the goal is to attain optimal-ity of this function, in the space defined by the modelparameters and other intermediate variables. Moreover,these algorithms all bear a common style, in that they re-sort to an iterative-convergent procedure (see Eq. 1). It isnoteworthy that iterative-convergent computing tasks arevastly different from conventional programmatic comput-ing tasks (such as database queries and keyword extrac-tion), which reach correct solutions only if every deter-ministic operation is correctly executed, and strong con-sistency is guaranteed on the intermediate program state— thus, operational objectives such as fault tolerance andstrong consistency are absolutely necessary. However, anML program’s true goal is fast, efficient convergence toan optimal solution, and we argue that fine-grained faulttolerance and strong consistency are but one vehicle toachieve this goal, and might not even be the most effi-cient one.

We present a new distributed ML framework, Petuum,built on an ML-centric optimization-theoretic principle,as opposed to various operational objectives explored ear-lier. We begin by formalizing ML algorithms as iterative-convergent programs, which encompass a large space ofmodern ML such as stochastic gradient descent, MCMCfor determining point estimates in latent variable mod-els [9], coordinate descent, variational methods for graph-ical models [11], proximal optimization for structuredsparsity problems [3], among others. To our knowledge,no existing ML platform has considered such a wide spec-trum of ML algorithms, which exhibit diverse represen-tation abstractions, model and data access patterns, andsynchronization and scheduling requirements. So whatare the shared properties across such a “zoo of ML algo-rithms”? We believe that the key lies in the recognition ofa clear dichotomy between data (which is conditionally in-dependent and persistent throughout the algorithm) andmodel (which is internally coupled, and is transient be-fore converging to an optimum). This inspires a simpleyet statistically-rooted bimodal approach to parallelism:data parallel and model parallel distribution and execu-tion of a big ML program over a cluster of machines.This data parallel, model parallel approach keenly exploitsthe unique statistical nature of ML algorithms, particu-larly the following three properties: (1) Error tolerance —iterative-convergent algorithms are often robust againstlimited errors in intermediate calculations; (2) Dynamicstructural dependency — during execution, the chang-ing correlation strengths between model parameters arecritical to efficient parallelization; (3) Non-uniform con-vergence — the number of steps required for a parameter

2

to converge can be highly skewed across parameters. Thecore goal of Petuum is to execute these iterative updatesin a manner that quickly converges to an optimum ofthe ML program’s objective function, by exploiting thesethree statistical properties of ML, which we argue arefundamental to efficient large-scale ML in cluster envi-ronments.

This design principle contrasts that of several existingframeworks discussed earlier. For example, central to theSpark framework [29] is the principle of perfect fault toler-ance and recovery, supported by a persistent memory ar-chitecture (Resilient Distributed Datasets); whereas cen-tral to the GraphLab framework is the principle of lo-cal and global consistency, supported by a vertex pro-gramming model (the Gather-Apply-Scatter abstraction).While these design principles reflect important aspects ofcorrect ML algorithm execution — e.g., atomic recover-ability of each computing step (Spark), or consistencysatisfaction for all subsets of model variables (GraphLab)— some other important aspects, such as the three sta-tistical properties discussed above, or perhaps ones thatcould be more fundamental and general, and which couldopen more room for efficient system designs, remain un-explored.

To exploit these properties, Petuum introduces threenovel system objectives grounded in the aforementionedkey properties of ML programs, in order to acceleratetheir convergence at scale: (1) Petuum synchronizes theparameter states with a bounded staleness guarantee,which achieves provably correct outcomes due to theerror-tolerant nature of ML, but at a much cheaper com-munication cost than conventional per-iteration bulk syn-chronization; (2) Petuum offers dynamic scheduling poli-cies that take into account the changing structural depen-dencies between model parameters, so as to minimize par-allelization error and synchronization costs; and (3) Sinceparameters in ML programs exhibit non-uniform conver-gence costs (i.e. different numbers of updates required),Petuum prioritizes computation towards non-convergedmodel parameters, so as to achieve faster convergence.

To demonstrate this approach, we show how a data-parallel and a model-parallel algorithm can be imple-mented on Petuum, allowing them to scale to large modelsizes with improved algorithm convergence times. This isillustrated in Figure 1, where Petuum is able to solve arange of ML problems at reasonably large model scales,even on relatively modest clusters (10-100 machines) thatare within reach of most ML practitioners. The experi-ments section provides more detailed benchmarks on arange of ML programs: topic modeling, matrix factoriza-tion, deep learning, Lasso regression, and distance metriclearning. These algorithms are only a subset of the full

open-source Petuum ML library2, which includes morealgorithms not explored in this paper: random forests, K-means, sparse coding, MedLDA, SVM, multi-class logisticregression, with many others being actively developed forfuture releases.

2 Preliminaries: On Data andModel Parallelism

We begin with a principled formulation of iterative-convergent ML programs, which exposes a dichotomy ofdata and model, that inspires the parallel system archi-tecture (§3), algorithm design (§4), and theoretical analy-sis (§5) of Petuum. Consider the following programmaticview of ML as iterative-convergent programs, driven byan objective function:Iterative-Convergent ML Algorithm: Given dataD and model L (i.e., a fitness function such as RMSloss, likelihood, margin), a typical ML problem can begrounded as executing the following update equation it-eratively, until the model state (i.e., parameters and/orlatent variables) A reaches some stopping criteria:

A(t) = F (A(t−1),∆L(A(t−1), D)) (1)

where superscript (t) denotes iteration. The update func-tion ∆L() (which improves the loss L) performs compu-tation on data D and model state A, and outputs inter-mediate results to be aggregated by F (). For simplicity,in the rest of the paper we omit L in the subscript withthe understanding that all ML programs of our interesthere bear an explicit loss function that can be used tomonitor the quality of convergence and solution, as op-pose to heuristics or procedures not associated such a lossfunction.

In large-scale ML, both data D and model A can bevery large. Data-parallelism, in which data is dividedacross machines, is a common strategy for solving BigData problems, while model-parallelism, which divides theML model, is common for Big Models. Below, we discussthe (different) mathematical implications of each paral-lelism (see Fig. 2).

2.1 Data Parallelism

In data-parallel ML, the data D is partitioned and as-signed to computational workers (indexed by p = 1..P );we denote the p-th data partition by Dp. We assumethat the function ∆() can be applied to each of thesedata subsets independently, yielding a data-parallel up-date equation:

A(t) = F (A(t−1),∑Pp=1 ∆(A

(t−1), Dp)). (2)

2Petuum is available as open source at http://petuum.org.

3

http://petuum.org

Data Parallel Model Parallel

~✓t+1 = ~✓t + �f~✓(D)

�~✓(D1)

�~✓(D2) �~✓(D3)

�~✓(Dn)

D ⌘ {D1, D2, . . . , Dn}

�~✓1(D)

�~✓2(D) �~✓3(D)

�~✓k(D)

~✓ ⌘ [~✓ T1 , ~✓ T2 , . . . , ~✓ Tk }T

Di?Dj | ✓, 8i 6= j ~✓i 6? ~✓j | D, 9(i, j)Figure 2: The difference between data and model parallelism:data samples are always conditionally independent given themodel, but there are some model parameters that are not in-dependent of each other.

In this definition, we assume that the ∆() outputs areaggregated via summation, which is commonly seen instochastic gradient descent or sampling-based algorithms.For example, in distance metric learning problem whichis optimized with stochastic gradient descent (SGD), thedata pairs are partitioned over different workers, and theintermediate results (subgradients) are computed on eachpartition and are summed before applied to update themodel parameters. Other algorithms can also be ex-pressed in this form, such as variational EM algorithmsA(t) =

∑Pp=1 ∆(A

(t−1), Dp). Importantly, this additiveupdates property allows the updates ∆() to be aggregatedat each local worker before transmission over the network,which is crucial because CPUs can produce updates ∆()much faster than they can be (individually) transmittedover the network. Additive updates are the foundationfor a host of techniques to speed up data-parallel exe-cution, such as minibatch, asynchronous and bounded-asynchronous execution, and parameter servers. Key tothe validity of additivity of updates from different work-ers is the notion of independent and identically distributed(iid) data, which is assumed for many ML programs, andimplies that each parallel worker contributes “equally”(in a statistical sense) to the ML algorithm’s progress via∆(), no matter which data subset Dp it uses.

2.2 Model Parallelism

In model-parallel ML, the model A is partitioned andassigned to workers p = 1..P and updated therein inparallel, running update functions ∆(). Unlike data-parallelism, each update function ∆() also takes a

scheduling function S(t−1)p (), which restricts ∆() to op-

erate on a subset of the model parameters A:

A(t) = F(A(t−1), {∆(A(t−1), S(t−1)p (A(t−1)))}Pp=1

), (3)

where we have omitted the data D for brevity and clarity.

S(t−1)p () outputs a set of indices {j1, j2, . . . , }, so that ∆()

only performs updates on Aj1 , Aj2 , . . . — we refer to suchselection of model parameters as scheduling.

Unlike data-parallelism which enjoys iid data proper-ties, the model parameters Aj are not, in general, inde-pendent of each other (Figure 2), and it has been estab-lished that model-parallel algorithms can only be effectiveif the parallel updates are restricted to independent (orweakly-correlated) parameters [15, 2, 22, 17]. Hence, ourdefinition of model-parallelism includes a global schedul-ing mechanism that can select carefully-chosen parame-ters for parallel updating.

The scheduling function S() opens up a large designspace, such as fixed, randomized, or even dynamically-changing scheduling on the whole space, or a subset of,the model parameters. S() not only can provide safetyand correctness (e.g., by selecting independent parame-ters and thus minimize parallelization error), but can of-fer substantial speed-up (e.g., by prioritizing computationonto non-converged parameters). In the Lasso example,Petuum uses S() to select coefficients that are weakly cor-related (thus preventing divergence), while at the sametime prioritizing coefficients far from zero (which are morelikely to be non-converged).

2.3 Implementing Data-and Model-Parallel Programs

Data- and model-parallel programs are stateful, in thatthey continually update shared model parameters A.Thus, an ML platform needs to synchronize A acrossall running threads and processes, and this should bedone in a high-performance non-blocking manner thatstill guarantees convergence. Ideally, the platform shouldalso offer easy, global-variable-like access to A (as op-posed to cumbersome message-passing, or non-statefulMapReduce-like functional interfaces). If the program ismodel-parallel, it may require fine control over parame-ter scheduling to avoid non-convergence; such capabilityis not available in Hadoop, Spark nor GraphLab withoutcode modification. Hence, there is an opportunity to ad-dress these considerations via a platform tailored to data-and model-parallel ML.

3 Petuum –a Platform for Distributed ML

A core goal of Petuum is to allow practitioners to eas-ily implement data-parallel and model-parallel ML al-gorithms. Petuum provides APIs to key systems thatmake data- and model-parallel programming easier: (1)a parameter server system, which allows programmersto access global model state A from any machine viaa convenient distributed shared-memory interface thatresembles single-machine programming, and adopts a

4

Sched

Client

Sched

Client

Sched

Client

Scheduler

PS

Client

Worker

PS

Client

PS server

Sched

Client

PS serverWorker

PS

Client

….ML App Code ML App Code

Consistency

Controller

Consistency

Controller

Dependency/

Priority Mgr.

Network Layer

parameter exchange channel

scheduling control channel

Data Partition

Data Partition

ModelPartition

ModelPartition

SchedulingData

Figure 3: Petuum system: scheduler, workers, parameterservers.

bounded-asychronous consistency model that preservesdata-parallel convergence guarantees, thus freeing usersfrom explicit network synchronization; (2) a scheduler,which allows fine-grained control over the parallel order-ing of model-parallel updates ∆() — in essence, the sched-uler allows users to define their own ML application con-sistency rules.

3.1 Petuum System Design

ML algorithms exhibit several principles that can be ex-ploited to speed up distributed ML programs: depen-dency structures between parameters, non-uniform con-vergence of parameters, and a limited degree of error tol-erance [10, 4, 15, 30, 16, 17]. Petuum allows practitionersto write data-parallel and model-parallel ML programsthat exploit these principles, and can be scaled to BigData and Big Model applications. The Petuum systemcomprises three components (Fig. 3): scheduler, work-ers, and parameter server, and Petuum ML programs arewritten in C++ (with Java support coming in the nearfuture).Scheduler: The scheduler system enables model-parallelism, by allowing users to control which modelparameters are updated by worker machines. This isperformed through a user-defined scheduling function

schedule() (corresponding to S(t−1)p ()), which outputs

a set of parameters for each worker — for example, asimple schedule might pick a random parameter for everyworker, while a more complex scheduler (as we will show)may pick parameters according to multiple criteria, suchas pair-wise independence or distance from convergence.The scheduler sends the identities of these parametersto workers via the scheduling control channel (Fig. 3),while the actual parameter values are delivered througha parameter server system that we will soon explain; thescheduler is responsible only for deciding which parame-ters to update. Later, we will discuss some of the theo-retical guarantees enjoyed by model-parallel schedules.

Several common patterns for schedule design are worthhighlighting: the simplest option is a fixed-schedule(schedule fix()), which dispatches model parametersA in a pre-determined order (as is common in exist-

ing ML algorithm implementations). Static, round-robinschedules (e.g. repeatedly loop over all parameters) fitthe schedule fix() model. Another type of sched-ule is dependency-aware (schedule dep()) schedul-ing, which allows re-ordering of variable/parameter up-dates to accelerate model-parallel ML algorithms such asLasso regression. This type of schedule analyzes the de-pendency structure over model parameters A, in orderto determine their best parallel execution order. Finally,prioritized scheduling (schedule pri()) exploits un-even convergence in ML, by prioritizing subsets of vari-ables Usub ⊂ A according to algorithm-specific criteria,such as the magnitude of each parameter, or boundaryconditions such as KKT.

Because scheduling functions schedule() may becompute-intensive, Petuum uses pipelining to overlapscheduling computations schedule() with worker execu-tion, so workers are always doing useful computation. Inaddition, the scheduler is responsible for central aggrega-tion via the pull() function (corresponding to F ()), if itis needed.Workers: Each worker p receives parameters to be up-dated from the scheduler function schedule(), and thenruns parallel update functions push() (corresponding to∆()) on data D. Petuum intentionally does not spec-ify a data abstraction, so that any data storage systemmay be used — workers may read from data loaded intomemory, or from disk, or over a distributed file system ordatabase such as HDFS. Furthermore, workers may touchthe data in any order desired by the programmer: in data-parallel stochastic algorithms, workers might sample onedata point at a time, while in batch algorithms, workersmight instead pass through all data points in one itera-tion. While push() is being executed, the model state Ais automatically synchronized with the parameter servervia the parameter exchange channel, using a distributedshared memory programming interface that convenientlyresembles single-machine programming. After the work-ers finish push(), the scheduler may use the new modelstate to generate future scheduling decisions.Parameter Server: The parameter server (PS) pro-vides global access to model parameters A, via a con-venient distributed shared memory API that is similarto table-based or key-value stores. To take advantage ofML-algorithmic principles, the PS implements the StaleSynchronous Parallel (SSP) consistency model [10, 4],which reduces network synchronization and communica-tion costs, while maintaining bounded-staleness conver-gence guarantees implied by SSP. We will discuss theseguarantees in more detail later.

3.2 Programming Interface

Figure 4 shows a basic Petuum program, consisting ofa central scheduler function schedule(), a parallel up-

5

// Petuum Program Structure

schedule() {// This is the (optional) scheduling function// It is executed on the scheduler machinesA_local = PS.get(A) // Parameter server readPS.inc(A,change) // Can write to PS here if needed// Choose variables for push() and returnsvars = my_scheduling(DATA,A_local)return svars

}

push(p = worker_id(), svars = schedule()) {// This is the parallel update function// It is executed on each of P worker machinesA_local = PS.get(A) // Parameter server read// Perform computation and send return values to pull()// Or just write directly to PSchange1 = my_update1(DATA,p,A_local)change2 = my_update2(DATA,p,A_local)PS.inc(A,change1) // Parameter server incrementreturn change2

}

pull(svars = schedule(), updates = (push(1), ..., push(P)) ) {// This is the (optional) aggregation function// It is executed on the scheduler machinesA_local = PS.get(A) // Parameter server read// Aggregate updates from push(1..P) and write to PSmy_aggregate(A_local,updates)PS.put(A,change) // Parameter server overwrite

}

Figure 4: Petuum Program Structure.

date function push(), and a central aggregation functionpull(). The model variables A are held in the parameterserver, which can be accessed at any time from any func-tion via the PS object. The PS object can be accessed fromany function, and has 3 functions: PS.get() to read a pa-rameter, PS.inc() to add to a parameter, and PS.put()to overwrite a parameter. With just these operations,the SSP consistency model automatically ensures param-eter consistency between all Petuum components; no ad-ditional user programming is necessary. Finally, we useDATA to represent the data D; as noted earlier, this canbe any 3rd-party data structure, database, or distributedfile system.

4 Petuum Parallel Algorithms

Now we turn to development of parallel algorithms forlarge-scale distributed ML problems, in light of the dataand model parallel principles underlying Petuum. We fo-cus on a new data-parallel Distance Metric Learning al-gorithm, and a new model-parallel Lasso algorithm, butour strategies apply to a broad spectrum of other MLproblems as briefly discussed at the end of this section.We show that with the Petuum system framework, wecan easily realize these algorithms on distributed clusterswithout dwelling on low level system programming, ornon-trivial recasting of our ML problems into represen-tations such as RDDs or vertex programs. Instead ourML problems can be coded at a high level, more akin toMatlab or R.

4.1 Data-Parallel Distance Metric Learn-ing

Let us first consider a large-scale Distance Metric Learn-ing (DML) problem. DML improves the performance ofother ML programs such as clustering, by allowing do-main experts to incorporate prior knowledge of the form“data points x, y are similar (or dissimilar)” [26] — forexample, we could enforce that “books about science aredifferent from books about art”. The output is a distancefunction d(x, y) that captures the aforementioned priorknowledge. Learning a proper distance metric [5, 26] is es-sential for many distance based data mining and machinelearning algorithms, such as retrieval, k-means clusteringand k-nearest neighbor (k-NN) classification. DML hasnot received much attention in the Big Data setting, andwe are not aware of any distributed implementations ofDML.

The most popular version of DML tries to learn a Ma-halanobis distance matrix M (symmetric and positive-semidefinite), which can then be used to measure the dis-tance between two samples D(x, y) = (x− y)TM(x− y).Given a set of “similar” sample pairs S = {(xi, yi)}|S|i=1,and a set of “dissimilar” pairs D = {(xi, yi)}|D|i=1, DMLlearns the Mahalanobis distance by optimizing

minM∑

(x,y)∈S(x− y)TM(x− y)

s.t. (x− y)TM(x− y) ≥ 1,∀(x, y) ∈ DM � 0

(4)

where M � 0 denotes that M is required to be positivesemidefinite. This optimization problem tries to minimizethe Mahalanobis distances between all pairs labeled assimilar while separating dissimilar pairs with a margin of1.

In its original form, this optimization problem is diffi-cult to parallelize due to the constraint set. To create adata-parallel optimization algorithm and implement it onPetuum, we shall relax the constraints via slack variables(similar to SVMs). First, we replace M with LTL, andintroduce slack variables ξ to relax the hard constraint inEq.(4), yielding

minL∑

(x,y)∈S‖L(x− y)‖2 + λ ∑

(x,y)∈Dξx,y

s.t. ‖L(x− y)‖2 ≥ 1− ξx,y, ξx,y ≥ 0,∀(x, y) ∈ D(5)

Using hinge loss, the constraint in Eq.(5) can be elimi-nated, yielding an unconstrained optimization problem:

minL∑

(x,y)∈S‖L(x− y)‖2

+λ∑

(x,y)∈Dmax(0, 1− ‖L(x− y)‖2) (6)

Unlike the original constrained DML problem, this relax-ation is fully data-parallel, because it now treats the dis-

6

// Data-Parallel Distance Metric Learning

schedule() { // Empty, do nothing }

push() {L_local = PS.get(L) // Bounded-async read from param serverchange = 0for c=1..C // Minibatch size C

(x,y) = draw_similar_pair(DATA)(a,b) = draw_dissimilar_pair(DATA)change += DeltaL(L_local,x,y,a,b) // SGD from Eq 7

PS.inc(L,change/C) // Add gradient to param server}

pull() { // Empty, do nothing }

Figure 5: Petuum DML data-parallel pseudocode.

similar pairs as iid data to the loss function (just like thesimilar pairs); hence, it can be solved via data-parallelStochastic Gradient Descent (SGD). SGD can be natu-rally parallelized over data, and we partition the datapairs onto P machines. Every iteration, each machinep randomly samples a minibatch of similar pairs Sp anddissimilar pairs Dp from its data shard, and computes thefollowing update to L:

4Lp =∑

(x,y)∈Sp 2L(x− y)(x− y)T− ∑(a,b)∈Dp 2L(a− b)(a− b)T · I(‖L(a− b)‖2 ≤ 1)

(7)where I(·) is the indicator function.

Figure 5 shows pseudocode for Petuum DML, which issimple to implement because the parameter server sys-tem PS abstracts away complex networking code under asimple get()/read() API. Moreover, the PS automati-cally ensures high-throughput execution, via a bounded-asynchronous consistency model (Stale Synchronous Par-allel) that can provide workers with stale local copies ofthe parameters L, instead of forcing workers to wait fornetwork communication. Later, we will review the strongconsistency and convergence guarantees provided by theSSP model.

Since DML is a data-parallel algorithm, only the par-allel update push() needs to be implemented (Figure 5).The scheduling function schedule() is empty (becauseevery worker touches every model parameter L), and wedo not need aggregation push() for this SGD algorithm.In our next example, we will show how schedule() andpush() can be used to implement model-parallel execu-tion.

4.2 Model-Parallel Lasso

Lasso is a widely used model to select features in high-dimensional problems, such as gene-disease associationstudies, or in online advertising via `1-penalized regres-sion [8]. Lasso takes the form of an optimization problem:

minβ`(X,y,β) + λ

∑j

|βj |, (8)

// Model-Parallel Lasso

schedule() {for j=1..J // Update priorities for all coeffs beta_j

c_j = square(beta_j) + eta // Magnitude prioritization(s_1, ..., s_L’) = random_draw(distribution(c_1, ..., c_J))// Choose L

Lasso. We use schedule() to choose parameters withlow dependency, and to prioritize non-converged pa-rameters. Petuum pipelines schedule() and push();thus schedule() does not slow down workers runningpush(). Furthermore, by separating the scheduling codeschedule() from the core optimization code push() andpull(), Petuum makes it easy to experiment with com-plex scheduling policies that involve prioritization anddependency checking, thus facilitating the implementa-tion of new model-parallel algorithms — for example,one could use schedule() to prioritize according toKKT conditions in a constrained optimization problem,or to perform graph-based dependency checking like inGraphlab [17]. Later, we will show that the above Lassoschedule schedule() is guaranteed to converge, and givesus near optimal solutions by controlling errors from par-allel execution. The pseudocode for scheduled model par-allel Lasso under Petuum is shown in Figure 6.

4.3 Other Algorithms

We have implemented other data- and model-parallel al-gorithms on Petuum as well. Here, we briefly mention afew, while noting that many others are included in thePetuum open-source library.

Topic Model (LDA): For LDA, the key parameteris the “word-topic” table, that needs to be updated byall worker machines. We adopt a simultaneous data-and-model-parallel approach to LDA, and use a fixed schedulefunction schedule fix() to cycle disjoint subsets of theword-topic table and data across machines for updating(via push() and pull()), without violating structuraldependencies in LDA.

Matrix Factorization (MF): High-rank decomposi-tions of large matrices for improved accuracy [32] can besolved by a model-parallel approach, and we implement itvia a fixed schedule function schedule fix(), where eachworker machine only performs the model update push()on a disjoint, unchanging subset of factor matrix rows.

Deep Learning (DL): We implemented two types onPetuum: a general-purpose fully-connected Deep NeuralNetwork (DNN) using the cross-entropy loss, and a Con-volutional Neural Network (CNN) for image classificationbased off the open-source Caffe project. We adopt a data-parallel strategy schedule fix(), where each worker usesits data subset to perform updates push() to the fullmodel A. While this data-parallel strategy could beamenable to MapReduce, Spark and GraphLab, we arenot aware of DL implementations on those platforms.

5 Principles and Theory

Our iterative-convergent formulation of ML programs,and the explicit notion of data and model parallelism,make it convenient to explore three key properties of

(c)(a) (b)

Figure 7: Key properties of ML algorithms: (a) Non-uniformconvergence; (b) Error-tolerant convergence; (c) Dependencystructures amongst variables.

ML programs — error-tolerant convergence, non-uniformconvergence, dependency structures (Fig. 7) — andto analyze how Petuum exploits these properties in atheoretically-sound manner to speed up ML programcompletion at Big Learning scales.

Some of these properties have previously been success-fully exploited by a number of bespoke, large-scale imple-mentations of popular ML algorithms: e.g. topic mod-els [28, 16], matrix factorization [27, 13], and deep learn-ing [14]. It is notable that MapReduce-style systems (suchas Hadoop [24] and Spark [29]) often do not fare compet-itively against these custom-built ML implementations,and one of the reasons is that these key ML properties aredifficult to exploit under a MapReduce-like abstraction.Other abstractions may offer a limited degree of oppor-tunity — for example, vertex programming [17] permitsgraph dependencies to influence model-parallel execution.

5.1 Error tolerant convergence

Data-parallel ML algorithms are often robust against mi-nor errors in intermediate calculations; as a consequence,they still execute correctly even when their model pa-rameters A experience synchronization delays (i.e. theP workers only see old or stale parameters), providedthose delays are strictly bounded [19, 10, 4, 33, 1, 12].Petuum exploits this error-tolerance to reduce networkcommunication/synchronization overheads substantially,by implementing the Stale Synchronous Parallel (SSP)consistency model [10, 4] on top of the parameter serversystem, which provides all machines with access to theparameters A.

The SSP consistency model guarantees that if aworker reads from parameter server at iteration c, itis guaranteed to receive all updates from all workerscomputed at and before iteration c− s− 1, where s is thestaleness threshold. If this is impossible because somestraggling worker is more than s iterations behind, thereader will stop until the straggler catches up and sendsits updates. For stochastic gradient descent algorithms(such as the DML program), SSP has very attractivetheoretical properties [4], which we partially re-state here:

Theorem 1 (adapted from [4]) SGD under SSP,

convergence in probability: Let f(x) =∑Tt=1 ft(x)

be a convex function, where the ft are also convex. Wesearch for a minimizer x∗ via stochastic gradient descent

8

on each component ∇ft under SSP, with staleness pa-rameter s and P workers. Let ut := −ηt∇tft(x̃t) withηt =

η√t. Under suitable conditions (ft are L-Lipschitz

and bounded divergence D(x||x′) ≤ F 2), we have

P

[R [X]

T− 1√

T

(ηL2 +

F 2

η+ 2ηL2µγ

)≥ τ

]≤ exp

{ −Tτ22η̄Tσγ +

23ηL

2(2s+ 1)Pτ

}where R[X] :=

∑Tt=1 ft(x̃t) − f(x∗), and η̄T =

η2L4(lnT+1)T = o(T ).

This means that R[X]T converges to O(T−1/2) in probabil-

ity with an exponential tail-bound; convergence is fasterwhen the observed staleness average µγ and variance σγare smaller (and SSP ensures both µγ , σγ are as small aspossible). Dai et al. also showed that the variance of xcan be bounded, ensuring reliability and stability near anoptimum [4].

5.2 Dependency structures

Naive parallelization of model-parallel algorithms (e.g.coordinate descent) may lead to uncontrolled paral-lelization error and non-convergence, caused by inter-parameter dependencies in the model. Such dependen-cies have been thoroughly analyzed under fixed execu-tion schedules (where each worker updates the same setof parameters every iteration) [22, 2, 21], but there hasbeen little research on dynamic schedules that can react tochanging model dependencies or model state A. Petuum’sscheduler allows users to write dynamic scheduling func-

tions S(t)p (A(t)) — whose output is a set of model indices

{j1, j20, . . . }, telling worker p to update Aj1 , Aj2 , . . . —as per their application’s needs. This enables ML pro-grams to analyze dependencies at run time (implementedvia schedule()), and select subsets of independent (ornearly-independent) parameters for parallel updates.

To motivate this, we consider a generic optimizationproblem, which many regularized regression problems —including the Petuum Lasso example — fit into:

minw∈Rd

f(w) + r(w), (10)

where r(w) =∑i r(wi) is separable and f has β-Lipschitz

continuous gradient in the following sense:

f(w + z) ≤ f(w) + z>∇f(w) + β2 z>X>Xz, (11)

where X = [x1, . . . ,xd] are d feature vectors. W.l.o.g.,we assume that each feature vector xi is normalized, i.e.,‖xi‖2 = 1, i = 1, . . . , d. Therefore |x>i xj | ≤ 1 for all i, j.

In the regression setting, f(w) represents a least-squares loss, r(w) represents a separable regularizer (e.g.`1 penalty), and xi represents the i-th feature column ofthe design (data) matrix, each element in xi is a separatedata sample. In particular, |x>i xj | is the correlation be-tween the i-th and j-th feature columns. The parametersw are simply the regression coefficients.

In the context of the model-parallel equation (3), wecan map the model A = w, the data D = X, and theupdate equation ∆(A,Sp(A)) to

w+jp ← arg minz∈Rβ2 [z − (wjp − 1β gjp)]2 + r(z), (12)

where S(t)p (A) has selected a single coordinate jp to be

updated by worker p — thus, P coordinates are updatedin every iteration. The aggregation function F () simplyallows each update wjp to pass through without change.

The effectiveness of parallel coordinate descent depends

on how the schedule S(t)p () selects the coordinates jp. In

particular, naive random selection can lead to poor con-vergence rate or even divergence, with error proportionalto the correlation |x>jaxjb | between the randomly-selectedcoordinates ja, jb [22, 2]. An effective and cheaply-

computable schedule S(t)RRP,p() involves randomly propos-

ing a small set of Q > P features {j1, . . . , jQ}, and thenfinding P features in this set such that |x>jaxjb | ≤ θ forsome threshold θ, where ja, jb are any two features inthe set of P . This requires at most O(B2) evaluations of|x>jaxjb | ≤ θ (if we cannot find P features that meet thecriteria, we simply reduce the degree of parallelism). Wehave the following convergence theorem:

Theorem 2 SRRP () convergence: Let k = 1 and � :=d(P−1)(ρ−1)B(B−1)τ2 ≈

(P−1)(ρ−1)d < 1, for constants d,B. After t

iterations,

E[F (w(t))− F (w?)] ≤ CdβP (1− �)

1

t, (13)

where F (w) := f(w)+r(w) and w? is a minimizer of F .

For reference, the Petuum Lasso scheduler uses SRRP (),augmented with a prioritizer we will describe soon.

In addition to asymptotic convergence, we show thatSRRP ’s trajectory is close to ideal parallel execution:

Theorem 3 SRRP () is close to ideal execution: LetSideal() be an oracle schedule that always proposes P ran-

dom features with zero correlation. Let w(t)ideal be its pa-

rameter trajectory, and let w(t)RRP be the parameter tra-

jectory of SRRP (). Then,

E[|w(t)ideal −w(t)RRP |] ≤

2JPm

(T + 1)2P̂L2XTXC, (14)

for constants C,m,L, P̂ .

9

The proofs for both theorems can be found in the onlinesupplement3.

SRRP () is different from Scherrer et al. [22], who pre-cluster all M features before starting coordinate descent,in order to find “blocks” of nearly-independent parame-ters. In the Big Data and especially Big Model setting,feature clustering can be prohibitive — fundamentally, itrequires O(M2) evaluations of |x>i xj | for all M2 featurecombinations (i, j), and although greedy clustering algo-rithms can mitigate this to some extent, feature clusteringis still impractical when M is very large, as seen in someregression problems [8]. The proposed SRRP () only needsto evaluate a small number of |x>i xj | every iteration, andwe explain next, the random selection can be replacedwith prioritization to exploit non-uniform convergence inML problems.

5.3 Non-uniform convergence

In model-parallel ML programs, it has been empiricallyobserved that some parameters Aj can converge in muchfewer/more updates than other parameters [15]. Forinstance, this happens in Lasso regression because themodel enforces sparsity, so most parameters remain atzero throughout the algorithm, with low probability of be-coming non-zero again. Prioritizing Lasso parameters ac-cording to their magnitude greatly improves convergenceper iteration, by avoiding frequent (and wasteful) updatesto zero parameters [15].

We call this non-uniform ML convergence, which can

be exploited via a dynamic scheduling function S(t)p (A(t))

whose output changes according to the iteration t — forinstance, we can write a scheduler Smag() that proposesparameters with probability proportional to their current

magnitude (A(t)j )

2. This Smag() can be combined withthe earlier dependency structure checking, leading to adependency-aware, prioritizing scheduler. Unlike the de-pendency structure issue, prioritization has not receivedas much attention in the ML literature, though it hasbeen used to speed up the PageRank algorithm, which isiterative-convergent [31].

The prioritizing schedule Smag() can be analyzed inthe context of the Lasso problem. First, we rewriteit by duplicating original J features with oppositesign: F (β) := minβ

12 ‖y−Xβ‖

22 + λ

∑2Jj=1 βj . Here, X

contains 2J features and βj ≥ 0, for all j = 1, . . . , 2J .

Theorem 4 (Adapted from [15]) Optimality ofLasso priority scheduler: Suppose B is the set ofindices of coefficients updated in parallel at the t-th it-eration, and ρ is sufficiently small constant such that

ρδβ(t)j δβ

(t)k ≈ 0, for all j 6= k ∈ B. Then, the sampling

3http://petuum.github.io/papers/kdd15_supp.pdf

0.65  0.7  0.75  0.8  0.85  0.9  0.95  

1  1.05  

0   200   400   600   800   1000  

Objec&v

e  

Time  (second)  

Petuum  Distance  Metric  Learning  

1  machine  

2  machines  

3  machines  

4  machines  

0.05  0.07  0.09  0.11  0.13  0.15  0.17  0.19  0.21  

0   200   400   600   800   1000   1200   1400  

Objec&v

e  

Time  (second)  

Lasso  

Petuum  Lasso  

Shotgun  

Figure 8: Left: Petuum DML convergence curve with differ-ent number of machines from 1 to 4. Right: Lasso conver-gence curve by Petumm Lasso and Shotgun.

distribution p(j) ∝ (δβ(t)j )2 approximately maximizes alower bound on EB[F (β

(t))− F (β(t) + ∆β(t))].

This theorem shows that a prioritizing scheduler speedsup Lasso convergence by decreasing the objective as muchas possible every iteration. The pipelined Petuum sched-

uler system approximates p(j) ∝ (δβ(t)j )2 with p′(j) ∝δ(β

(t−1)j )

2 + η, because δβ(t)j is unavailable until all com-

putations on β(t)j have finished (and we want schedule

before that happens, so that workers are fully occupied).Since we are approximating, we add a constant η to en-sure all βj ’s have a non-zero probability of being updated.

6 Performance

Petuum’s ML-centric system design supports a variety ofML programs, and improves their performance on BigData in the following senses: (1) Petuum implementa-tions of DML and Lasso achieve significantly faster con-vergence rate than baselines (i.e., DML implemented onsingle machine, and Shotgun [2]); (2) Petuum ML im-plementations can run faster than other platforms (e.g.Spark, GraphLab4), because Petuum can exploit modeldependencies, uneven convergence and error tolerance;(3) Petuum ML implementations can reach larger modelsizes than other platforms, because Petuum stores MLprogram variables in a lightweight fashion (on the param-eter server and scheduler); (4) for ML programs withoutdistributed implementations, we can implement them onPetuum and show good scaling with an increasing num-ber of machines. We emphasize that Petuum is, for themoment, primarily about allowing ML practitioners toimplement and experiment with new data/model-parallelML algorithms on small-to-medium clusters; Petuum cur-rently lacks features that are necessary for clusters with≥ 1000 machines, such as automatic recovery from ma-chine failure. Our experiments are therefore focused onclusters with 10-100 machines, in accordance with ourtarget users.

Performance of Distance Metric Learning andLasso

4We omit Hadoop, as it is well-established that Spark andGraphLab significantly outperform it [29, 17].

10

http://petuum.github.io/papers/kdd15_supp.pdf

0

1

2

3

4

5

6

LDA Matrix Fact.

Re

lati

ve S

pe

ed

up

Platforms vs Petuum

Pe

tuu

m

Gra

ph

Lab

(o

ut

of

me

mo

ry)

Pet

uu

m

Yah

oo

LDA

Gra

ph

Lab

(o

ut

of

mem

ory

)

Spar

k 0

0.5

1

1.5

2

2.5

3

3.5

4

Deep Learning: CNN Distance Metric Learning

Re

lati

ve S

pe

ed

up

Single-machine vs Petuum Ideal Linear Speedup

Pe

tuu

m C

affe

4 m

ach

Ori

gin

al C

affe

1 m

ach

Pe

tuu

m D

ML

4 m

ach

Xin

g20

02

1 m

ach

Figure 9: Left: Petuum performance: relative speedup vspopular platforms (larger is better). Across ML programs,Petuum is at least 2-10 times faster than popular implemen-tations. Right: Petuum is a good platform for writing clus-ter versions of existing single-machine algorithms, achievingnear-linear speedup with increasing number of machines (CaffeCNN and DML).

We first demonstrate the performance of DML andlasso, implemented under Petuum. In Figure 8, we show-case the convergence of Petuum and baselines using afixed model size (we used a 21504 × 1000 distance ma-trix for DML; 100M features for Lasso). For DML, in-creasing the number of machines consistently increasesthe convergence speed. Petuum DML achieves 3.8 timesspeedup with 4 machines and 1.9 times speedup with 2machines, demonstrating that Petuum DML has the po-tential to scale very well with more machines. For Lasso,given the same number of machines, Petuum achieved asignificantly faster convergence rate than Shotgun (whichrandomly selects a subset of parameters to be updated).In the initial stage, Petuum lasso and Shotgun show sim-ilar convergence rates because Petuum updates every pa-rameter in the first iteration to “bootstrap” the scheduler(at least one iteration is required to initialize all param-eters). After this initial stage, Petuum dramatically de-creases the Lasso objective compared to Shotgun, by tak-ing advantage of dependency structures and non-uniformconvergence via the scheduler.

Platform Comparison Figure 9 (left) comparesPetuum to popular ML platforms (Spark and GraphLab)and well-known cluster implementations (YahooLDA).For two common ML programs of LDA and MF, weshow the relative speedup of Petuum over the other plat-forms’ implementations. In general, Petuum is between2-6 times faster than other platforms; the differences helpto illustrate the various data/model-parallel features inPetuum. For MF, Petuum uses the same model-parallelapproach as Spark and GraphLab, but it performs twiceas fast as Spark, while GraphLab ran out of memory.On the other hand, Petuum LDA is nearly 6 times fasterthan YahooLDA; the speedup mostly comes from schedul-ing S(), which enables correct, dependency-aware model-parallel execution.

Scaling to Larger Models

Here, we show that Petuum supports larger ML modelsfor the same amount of cluster memory. Figure 10 shows

Figure 10: Left: LDA convergence time: Petuum vs Ya-hooLDA (lower is better). Petuum’s data-and-model-parallelLDA converges faster than YahooLDA’s data-parallel-only im-plementation, and scales to more LDA parameters (largervocab size, number of topics). Right panels: Matrix Fac-torization convergence time: Petuum vs GraphLab vs Spark.Petuum is fastest and the most memory-efficient, and is theonly platform that could handle Big MF models with rankK ≥ 1000 on the given hardware budget.

ML program running time versus model size, given a fixednumber of machines — the left panel compares PetuumLDA and YahooLDA; PetuumLDA converges faster andsupports LDA models that are > 10 times larger5, al-lowing long-tail topics to be captured. The right pan-els compare Petuum MF versus Spark and GraphLab;again Petuum is faster and supports much larger MFmodels (higher rank) than either baseline. Petuum’smodel scalability is the result of two factors: (1) model-parallelism, which divides the model across machines; (2)a lightweight parameter server system with minimal stor-age overhead.

Fast Cluster Implementations of New ML Pro-grams

We show that Petuum facilitates the development ofnew ML programs without existing cluster implementa-tions. In Figure 9 (right), we present two instances: first,a cluster version of the open-source Caffe CNN toolkit,created by adding ∼ 600 lines of Petuum code. The basicdata-parallel strategy was left unchanged, so the Petuumport directly tests Petuum’s efficiency. Compared tothe original single-machine Caffe with no network com-munication, Petuum achieves approaching-linear speedup(3.1-times speedup on 4 machines) due to the parameterserver’s SSP consistency for managing network communi-cation. Second, we compare the Petuum DML programagainst the original DML algorithm proposed in [26] (de-noted by Xing2002), which is optimized with SGD on asingle-machine (with parallelization over matrix opera-tions). The intent is to show that a fairly simple data-parallel SGD implementation of DML (the Petuum pro-gram) can greatly speed up execution over a cluster. ThePetuum implementation converges 3.8 times faster thanXing2002 on 4 machines — this provides evidence thatPetuum enables data/model-parallel algorithms to be ef-ficiently implemented over clusters.

Experimental settings

5LDA model size is equal to vocab size times number of topics.

11

We used 3 clusters with varying specifications, demon-strating Petuum’s adaptability to different hardware:“Cluster-1” has machines with 2 AMD cores, 8GB RAM,1Gbps Ethernet; “Cluster-2” has machines with 64 AMDcores, 128GB RAM, 40Gbps Infiniband; “Cluster-3” hasmachines with 16 Intel cores, 128GB RAM, 10Gbps Eth-ernet.

LDA was run on 128 Cluster-1 nodes, using 3.9m En-glish Wikipedia abstracts with unigram (V = 2.5m) andbigram (V = 21.8m) vocabularies. MF and Lasso wererun on 10 Cluster-2 nodes, respectively using the Net-flix data and a synthetic Lasso dataset with N = 50ksamples and 100m features/parameters. CNN was runon 4 Cluster-3 nodes, using a 250k subset of Imagenetwith 200 classes, and 1.3m model parameters. The DMLexperiment was run on 4 Cluster-2 nodes, using the 1-million-sample Imagenet [7] dataset with 1000 classes(220m model parameters), and 200m similar/dissimilarstatements.

References

[1] A. Agarwal and J. C. Duchi. Distributed delayed stochas-tic optimization. In NIPS, 2011.

[2] J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin.Parallel coordinate descent for l1-regularized loss mini-mization. In ICML, 2011.

[3] X. Chen, Q. Lin, S. Kim, J. Carbonell, and E. Xing.Smoothing proximal gradient method for general struc-tured sparse learning. In UAI, 2011.

[4] W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, and E. P.Xing. High-performance distributed ml at scale throughparameter server consistency models. In AAAI. 2015.

[5] J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon.Information-theoretic metric learning. In Proceedings ofthe 24th international conference on Machine learning,pages 209–216. ACM, 2007.

[6] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin,Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker,K. Yang, and A. Ng. Large scale distributed deep net-works. In NIPS 2012, 2012.

[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, andL. Fei-Fei. Imagenet: A large-scale hierarchical imagedatabase. In Computer Vision and Pattern Recognition,2009. CVPR 2009. IEEE Conference on, pages 248–255.IEEE, 2009.

[8] H. B. M. et. al. Ad click prediction: a view from thetrenches. In KDD, 2013.

[9] T. L. Griffiths and M. Steyvers. Finding scientific topics.PNAS, 101(Suppl 1):5228–5235, 2004.

[10] Q. Ho, J. Cipar, H. Cui, J.-K. Kim, S. Lee, P. B. Gibbons,G. Gibson, G. R. Ganger, and E. P. Xing. More effectivedistributed ml via a stale synchronous parallel parameterserver. In NIPS, 2013.

[11] M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley.Stochastic variational inference. JMLR, 14, 2013.

[12] A. Kumar, A. Beutel, Q. Ho, and E. P. Xing. Fugue:Slow-worker-agnostic distributed learning for big modelson big data. In AISTATS.

[13] A. Kumar, A. Beutel, Q. Ho, and E. P. Xing. Fugue:Slow-worker-agnostic distributed learning for big modelson big data. In Proceedings of the Seventeenth Interna-tional Conference on Artificial Intelligence and Statistics,pages 531–539, 2014.

[14] Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen,G. Corrado, J. Dean, and A. Ng. Building high-level fea-tures using large scale unsupervised learning. In ICML,2012.

[15] S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. Gibson, and E. P.Xing. On model parallelism and scheduling strategies fordistributed machine learning. In NIPS. 2014.

[16] M. Li, D. G. Andersen, J. W. Park, A. J. Smola,A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, andB.-Y. Su. Scaling distributed machine learning with theparameter server. In OSDI, 2014.

[17] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin,and J. M. Hellerstein. Distributed GraphLab: A Frame-work for Machine Learning and Data Mining in theCloud. PVLDB, 2012.

[18] G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert,I. Horn, N. Leiser, and G. Czajkowski. Pregel: a sys-tem for large-scale graph processing. In ACM SIGMODInternational Conference on Management of data. ACM,2010.

[19] F. Niu, B. Recht, C. Ré, and S. J. Wright. Hogwild!:A lock-free approach to parallelizing stochastic gradientdescent. In NIPS, 2011.

[20] R. Power and J. Li. Piccolo: building fast, distributedprograms with partitioned tables. In OSDI. USENIX As-sociation, 2010.

[21] P. Richtárik and M. Takáč. Parallel coordinate de-scent methods for big data optimization. arXiv preprintarXiv:1212.0873, 2012.

[22] C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin.Feature clustering for accelerating parallel coordinate de-scent. NIPS, 2012.

[23] Y. Wang, X. Zhao, Z. Sun, H. Yan, L. Wang, Z. Jin,L. Wang, Y. Gao, J. Zeng, Q. Yang, et al. Towards topicmodeling for big data. arXiv preprint arXiv:1405.4402,2014.

[24] T. White. Hadoop: The definitive guide. O’Reilly Media,Inc., 2012.

[25] S. A. Williamson, A. Dubey, and E. P. Xing. Paral-lel markov chain monte carlo for nonparametric mixturemodels. In ICML, 2013.

[26] E. P. Xing, M. I. Jordan, S. Russell, and A. Y. Ng. Dis-tance metric learning with application to clustering withside-information. In Advances in neural information pro-cessing systems, pages 505–512, 2002.

12

[27] H.-F. Yu, C.-J. Hsieh, S. Si, and I. Dhillon. Scalable coor-dinate descent approaches to parallel matrix factorizationfor recommender systems. In Data Mining (ICDM), 2012IEEE 12th International Conference on, pages 765–774.IEEE, 2012.

[28] J. Yuan, F. Gao, Q. Ho, W. Dai, J. Wei, X. Zheng, E. P.Xing, T.-Y. Liu, and W.-Y. Ma. Lightlda: Big topicmodels on modest compute clusters. In Accepted to In-ternational World Wide Web Conference. 2015.

[29] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker,and I. Stoica. Spark: cluster computing with workingsets. In HotCloud, 2010.

[30] Y. Zhang, Q. Gao, L. Gao, and C. Wang. Priter: Adistributed framework for prioritized iterative computa-tions. In SOCC, 2011.

[31] Y. Zhang, Q. Gao, L. Gao, and C. Wang. Priter: Adistributed framework for prioritizing iterative computa-tions. Parallel and Distributed Systems, IEEE Transac-tions on, 24(9):1884–1893, 2013.

[32] Y. Zhou, D. Wilkinson, R. Schreiber, and R. Pan. Large-scale parallel collaborative filtering for the netflix prize.In Algorithmic Aspects in Information and Management,2008.

[33] M. Zinkevich, J. Langford, and A. J. Smola. Slow learnersare fast. In NIPS, 2009.

A Proof of Theorem 2

We prove that the Petuum SRRP () scheduler makes the Reg-ularized Regression Problem converge. We note that SRRP ()has the following properties: (1) the scheduler uniformly ran-domly selects Q out of d coordinates (where d is the number offeatures); (2) the scheduler performs dependency checking andretains P out of Q coordinates; (3) in parallel, each of the Pworkers is assigned one coordinate, and performs coordinatedescent on it:

w+jp ← arg minz∈Rβ2

[z − (wjp − 1β gjp)]2 + r(z), (15)

where gj = ∇jf(w) is the j-th partial derivative, and thecoordinate jp is assigned to the p-th worker. Note that (15) issimply the gradient update: w ← w− 1

βg, followed by applying

the proximity operator of r.

One way for the scheduler to select P coordinates into Q isto perform dependency checking: Coordinates i and j are inthe same block iff |x>i xj | ≤ θ for some parameter θ ∈ (0, 1).Consider the following matrix

∀i, Aii = 1, ∀i 6= j, Aij =

{x>i xj , if |x>i xj | ≤ θ0, otherwise

, (16)

whose spectral radius ρ = ρ(A) will play a major role in ouranalysis. A trivial bound for the spectral radius ρ(A) is:

|ρ− 1| ≤∑j 6=i

|Aij | ≤ (d− 1)θ. (17)

Thus, if θ is small, the spectral radius ρ is small.

Denote N the total number of pairs (i, j) that can passthe dependency check. Roughly N ∼ O(d2) if θ is close to 1(i.e., all possible pairs). We assume that each of such pair willbe selected by the scheduler with equal probability (i.e., 1/N).This can be achieved by rejection sampling. As a consequence,P , the number of coordinates selected by the scheduler, is arandom variable and may vary from step to step. In practice,we assume that P is equal to the number of available workers.

Theorem 2 Let � := d(EP2/EP−1)(ρ−1)

N≈ (EP−1)(ρ−1)

d< 1,

then after t steps, we have

E[F (wt)− F (w?)] ≤Cdβ

EP (1− �)1

t, (18)

where F (w) := f(w) + r(w) and w? denotes a (global) mini-mizer of F (whose existence is assumed for simplicity).

Proof of Theorem 2 We first bound the algorithm’sprogress at step t. To avoid cumbersome double indices, let

13

w = wt and z = wt+1, then applying (11)

E[F (z)− F (w)]

≤ E[ P∑p=1

gjp(w+jp− wjp) + r(w

+jp

)− r(wjp)

+β

2(w+jp − wjp)

2 +β

2

∑p6=q

(w+jp − wjp)(w+jq− wjq )x

>jpxjq

]

=EP

d

[g>(w+ −w) + r(w+)− r(w) + β

2‖w+ −w‖22

]+

βE[P (P − 1)]2N

(w+ −w)>(A− I)(w+ −w)

≤ −βEP2d‖w+ −w‖22 +

βE[P (P − 1)](ρ− 1)2N

‖w+ −w‖22

≤ −βEP (1− �)2d

‖w+ −w‖22,

where we define � = d(EP2/EP−1)(ρ−1)

N, and the second in-

equality follows from the optimality of w+ as defined in (15).Therefore as long as � < 1, the algorithm is decreasing theobjective. This in turn puts a limit on the expected num-ber of parallel workers P , roughly inverse proportional to thespectral radius ρ.

The rest of the proof follows the same line as that of shotgun[2]. To give a quick idea, consider the case where 0 ∈ ∂r(wt),then

F (wt+1)− F (w?) ≤ (wt+1 −w?)>g ≤ ‖wt+1 −w?‖2 · ‖g‖2,

and ‖wt+1 − wt‖22 = ‖g‖22/β2. Thus, defining δt = F (wt) −F (w?), we have

E(δt+1 − δt) ≤ −EP (1− �)

2dβ‖wt+1 −w?‖22E(δ2t+1) (19)

≤ − EP (1− �)2dβ‖wt+1 −w?‖22

[E(δt+1)]2. (20)

Using induction it follows that E(δt) ≤ CdβEP (1−�)1t

for someuniversal constant C. �

The theorem confirms some intuition: The bigger the ex-pected number of selected coordinates EP , the faster algo-rithm converges, but it also increases �, demonstrating a trade-off among parallelization and correctness. The variance EP 2

also plays a role here: the smaller it is, the faster the algo-rithm converges (since � is proportional to it). Of course, thebigger N is, i.e., less coordinates are correlated above θ, thefaster the algorithm converges (since � is inverse proportionalto it).

Remark: We compare Theorem 2 with Shotgun [2] andthe Block greedy algorithm in [22]. The convergence rate weget is similar to shotgun, but with a significant difference:Our spectral radius ρ = ρ(A) is potentially much smaller thanshotgun’s ρ(X>X), since by partitioning we zero out all en-tries in the correlation matrix X>X that are bigger than thethreshold θ. In other words, we get to control the spectralradius while shotgun is totally passive.

The convergence rate in [22] is CBP (1−�′)

1t, where �′ =

(P−1)(ρ′−1)B−1 . Compared with ours, we have a bigger (hence

worse) numerator (d vs. B) but the denominator (�′ vs. �)are not directly comparable: we have a bigger spectral ra-dius ρ and bigger d while [22] has a smaller spectral radius ρ′

(essentially taking a submatrix of our A) and smaller B − 1.Nevertheless, we note that [22] may have a higher per-stepcomplexity: each worker needs to check all of its assigned τcoordinates just to update one “optimal” coordinate. In con-trast, we simply pick a random coordinate, and hence can bemuch cheaper per-step.

B Proof of Theorem 3

For the Regularized Regression Problem, we prove that thePetuum SRRP () scheduler produces a solution trajectory

w(t)RRP that is close to ideal execution:

Theorem 3 (SRRP () is close to ideal execution) LetSideal() be an oracle schedule that always proposes P random

features with zero correlation. Let w(t)ideal be its parameter tra-

jectory, and let w(t)RRP be the parameter trajectory of SRRP ().

Then,

E[|w(t)ideal −w(t)RRP |] ≤

2JPm

(T + 1)2P̂L2XTXC, (21)

C is a data dependent constant, m is the strong convexityconstant, L is the domain width of Aj , and P̂ is the ex-pected number of indexes that SRRP () can actually parallelizein each iteration (since it may not be possible to find P nearly-independent parameters).

We assume that the objective function F (w) = f(w)+r(w)is strongly convex — for certain problems, this can be achievedthrough parameter replication, e.g. minw

12||y − Xw||22 +

λ∑2Mj=1 wj is the replicated form of Lasso regression seen in

Shotgun [2].

Lemma 1 The difference between successive updates is:

F (w+∆w)−F (w) ≤ −(∆w)T∆w+ 12

(∆w)TXTX∆w (22)

Proof: The Taylor expansion of F (w + ∆w) around w cou-

pled with the fact that F (w)′′′

(3rd-order) and higher orderderivatives are zero leads to the above result. �

Proof of Theorem 3 By using Lemma 1, and telescopingsum:

F (w(T )ideal)− F (w

(0)ideal) =

T∑t=1

−(∆w(t)ideal)>∆w

(t)ideal +

1

2(∆w

(t)ideal)

>X>X∆w(t)ideal (23)

Since Sideal chooses P features with 0 correlation,

F (w(T )ideal)− F (w

(0)ideal) =

T∑t=1

−(∆w(t)ideal)>∆w

(t)ideal

14

Again using Lemma 1, and telescoping sum:

F (w(T )RRP )− F (w

(0)RRP ) =

T∑t=1

−(∆w(t)RRP )>∆w

(t)RRP +

1

2(∆w

(t)RRP )

>X>X∆w(t)RRP

(24)

Taking the difference of the two sequences, we have:

F (w(T )ideal)− F (w

(T )RRP ) =(

T∑t=1

−(∆w(t)ideal)>∆w

(t)ideal

)

−

(T∑t=1

−(∆w(t)RRP )>∆w

(t)RRP +

1

2(∆w

(t)RRP )

>X>X∆w(t)RRP

)(25)

Taking expectations w.r.t. the randomness in iteration, in-dices chosen at each iteration, and the inherent randomnessin the two sequences, we have:

E[|F (w(T )ideal)− F (w(T )RRP )|] =

E[|(T∑t=1

−(∆w(t)ideal)T∆w

(t)ideal)

− (T∑t=1

−(∆w(t)RRP )T∆w

(t)RRP +

1

2(∆w

(t)RRP )

>X>X∆w(t)RRP )|]

= (Cdata +1

2)E[|

T∑t=1

(∆w(t)RRP )

>X>X∆w(t)RRP )|], (26)

where Cdata is a data dependent constant. Here, the differencebetween (∆w

(t)ideal)

>∆w(t)ideal and (∆w

(t)RRP )

>∆w(t)RRP can only

be possible due to (∆w(t)RRP )

>X>X∆w(t)RRP .

Following the proof in the shotgun paper [2], we get

E[|F (w(T )ideal)− F (w(T )RRP )|] ≤

2JP

(T + 1)2P̂L2XTXC, (27)

where C is a data dependent constant, L is the domain widthof wj (i.e. the difference between its maximum and minimumpossible values), and P̂ is the expected number of indexes thatSRRP () can actually parallelize in each iteration.

Finally, we apply the strong convexity assumption to get

E[|w(T )ideal −w(T )RRP |] ≤

2JPm

(T + 1)2P̂L2XTXC, (28)

where m is the strong convexity constant. �

15

1 Introduction2 Preliminaries: On Data and Model Parallelism2.1 Data Parallelism2.2 Model Parallelism2.3 Implementing Data- and Model-Parallel Programs

3 Petuum – a Platform for Distributed ML3.1 Petuum System Design3.2 Programming Interface

4 Petuum Parallel Algorithms4.1 Data-Parallel Distance Metric Learning4.2 Model-Parallel Lasso4.3 Other Algorithms

5 Principles and Theory5.1 Error tolerant convergence5.2 Dependency structures5.3 Non-uniform convergence

6 PerformanceA Proof of Theorem 2B Proof of Theorem 3