Name
Abstract | ||
|---|---|---|
Decentralized (i.e., serverless) training across edge nodes can suffer substantially from potential Byzantine nodes that can degrade the training performance. However, detection and mitigation of Byzantine behaviors in a decentralized learning setting is a daunting task, especially when the data distribution at the users is heterogeneous. As our main contribution, we propose
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
, a fast and computationally efficient Byzantine-robust algorithm for decentralized training systems, which leverages a novel sequential, memory-assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users. In the IID dataset setting, we provide the theoretical convergence guarantees of
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
, demonstrating its linear convergence rate. Furthermore, for the IID setting, we experimentally demonstrate that
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
is robust to various Byzantine attacks, including the strong Hidden attack, while providing up to absolute ~16% higher test accuracy over the state-of-the-art Byzantine-resilient decentralized learning approach. Additionally, we generalize
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
to the non-IID setting by proposing Anonymous Cyclic Data Sharing (ACDS), a technique that allows each node to anonymously share a random fraction of its local non-sensitive dataset (e.g., landmarks images) with all other nodes. Finally, to reduce the overall latency of
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
resulting from its sequential implementation over the logical ring, we propose
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil+</monospace>
that enables Byzantine-robust parallel training across groups of logical rings, and at the same time, it retains the performance gains of
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil</monospace>
due to sequential training within each group. Furthermore, we experimentally demonstrate the scalability gains of
<monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Basil+</monospace>
through different sets of experiments. |
Year | DOI | Venue |
|---|---|---|
2022 | 10.1109/JSAC.2022.3191347 | IEEE Journal on Selected Areas in Communications |
Keywords | DocType | Volume |
Decentralized training,federated learning,Byzantine-robustness | Journal | 40 |
Issue | ISSN | Citations |
9 | 0733-8716 | 0 |
PageRank | References | Authors |
0.34 | 5 | 3 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Ahmed Roushdy Elkordy | 1 | 0 | 0.34 |
| Saurav Prakash | 2 | 0 | 0.34 |
| Amir Salman Avestimehr | 3 | 1880 | 157.39 |