Name
Abstract | ||
|---|---|---|
Accelerators have emerged as an important component of modern cloud, datacenter, and HPC computing environments. However, launching tasks on remote accelerators across a network remains unwieldy, forcing programmers to send data in large chunks to amortize the transfer and launch overhead. By combining advances in intra-node accelerator unification with one-sided Remote Direct Memory Access (RDMA) communication primitives, it is possible to efficiently implement lightweight tasking across distributed-memory systems. This paper introduces Extended Task Queuing (XTQ), an RDMA-based active messaging mechanism for accelerators in distributed-memory systems. XTQ's direct NIC-to-accelerator communication decreases inter-node GPU task launch latency by 10-15% for small-to-medium sized messages and ameliorates CPU message servicing overheads. These benefits are shown in the context of MPI accumulate, reduce, and allreduce operations with up to 64 nodes. Finally, we illustrate how XTQ can improve the performance of popular deep learning workloads implemented in the Computational Network Toolkit (CNTK). |
Year | DOI | Venue |
|---|---|---|
2016 | 10.1109/SC.2016.79 | SC |
Keywords | Field | DocType |
Accelerator architectures,Computer architecture,Computer networks,Distributed computing,Network interfaces | Latency (engineering),Computer science,Computer network,Queueing theory,Artificial intelligence,Remote direct memory access,Deep learning,Distributed computing,Network interface,Central processing unit,Parallel computing,Schedule,Cloud computing | Conference |
ISBN | Citations | PageRank |
978-1-4673-8815-3 | 3 | 0.39 |
References | Authors | |
21 | 16 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Michael LeBeane | 1 | 34 | 5.28 |
| Brandon Potter | 2 | 3 | 0.39 |
| Abhisek Pan | 3 | 3 | 0.39 |
| Alexandru Dutu | 4 | 7 | 1.13 |
| Vinay Agarwala | 5 | 3 | 0.39 |
| Wonchan Lee | 6 | 7 | 1.85 |
| Deepak Majeti | 7 | 45 | 4.50 |
| Bibek Ghimire | 8 | 3 | 0.39 |
| Eric Van Tassell | 9 | 3 | 0.39 |
| Samuel Wasmundt | 10 | 3 | 0.39 |
| Brad Benton | 11 | 3 | 0.39 |
| Mauricio Breternitz, Jr. | 12 | 68 | 9.97 |
| Michael L. Chu | 13 | 292 | 13.12 |
| Mithuna Thottethodi | 14 | 789 | 55.35 |
| Lizy Kurian John | 15 | 2315 | 185.19 |
| Steven K. Reinhardt | 16 | 3885 | 226.69 |