Title | ||
---|---|---|
Lighter-Communication Distributed Machine Learning via Sufficient Factor Broadcasting. |
Abstract | ||
---|---|---|
Matrix-parametrized models (MPMs) are widely used in machine learning (ML) applications. In large-scale ML problems, the parameter matrix of a MPM can grow at an unexpected rate, resulting in high communication and parameter synchronization costs. To address this issue, we offer two contributions: first, we develop a computation model for a large family of MPMs, which share the following property: the parameter update computed on each data sample is a rank-1 matrix, i.e. the outer product of two \"sufficient factors\" (SFs). Second, we implement a decentralized, peer-to-peer system, Sufficient Factor Broadcasting (SFB), which broadcasts the SFs among worker machines, and reconstructs the update matrices locally at each worker. SFB takes advantage of small rank-1 matrix updates and efficient partial broadcasting strategies to dramatically improve communication efficiency. We propose a graph optimization based partial broadcasting scheme, which minimizes the delay of information dissemination under the constraint that each machine only communicates with a subset rather than all of machines. Furthermore, we provide theoretical analysis to show that SFB guarantees convergence of algorithms (under full broadcasting) without requiring a centralized synchronization mechanism. Experiments corroborate SFB's efficiency on four MPMs. |
Year | Venue | Field |
---|---|---|
2016 | UAI | Convergence (routing),Outer product,Broadcasting,Synchronization,Graph optimization,Sample (statistics),Computer science,Matrix (mathematics),Artificial intelligence,Machine learning,Computation,Distributed computing |
DocType | Citations | PageRank |
Conference | 3 | 0.37 |
References | Authors | |
8 | 7 |
Name | Order | Citations | PageRank |
---|---|---|---|
Pengtao Xie | 1 | 339 | 22.63 |
Jin Kyu Kim | 2 | 434 | 17.53 |
Yi Zhou | 3 | 65 | 17.55 |
Ho, Qirong | 4 | 636 | 30.75 |
Abhimanu Kumar | 5 | 227 | 9.76 |
Yaoliang Yu | 6 | 669 | 34.33 |
Bo Xing | 7 | 7332 | 471.43 |