Name
Abstract | ||
|---|---|---|
Batch normalization (BN) enables us to train various deep neural networks faster. However, the training accuracy will be significantly influenced with the decrease of input mini-batch size. To increase the model accuracy, a global mean and variance among all the input batch can be used, nevertheless communication across all devices is required in each BN layer, which reduces the training speed greatly. To address this problem, we propose progressive batch normalization, which can achieve a good balance between model accuracy and efficiency in multiple-GPU training. Experimental results show that our algorithm can obtain significant performance improvement over traditional BN without data synchronization across GPUs, achieving up to 18.4% improvement on training DeepLab for semantic segmentation task across 8 GPUs. |
Year | DOI | Venue |
|---|---|---|
2019 | 10.1007/s10766-018-0615-5 | International Journal of Parallel Programming |
Keywords | Field | DocType |
Batch normalization, Data parallelism, Deep learning | Normalization (statistics),Segmentation,Computer science,Parallel computing,Data synchronization,Data parallelism,Artificial intelligence,Deep learning,Computer engineering,Deep neural networks,Performance improvement | Journal |
Volume | Issue | ISSN |
47 | 3 | 1573-7640 |
Citations | PageRank | References |
0 | 0.34 | 8 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Lianke Qin | 1 | 0 | 0.34 |
| Yifan Gong | 2 | 1332 | 135.58 |
| Tianqi Tang | 3 | 342 | 19.66 |
| Yutian Wang | 4 | 0 | 4.06 |
| Jiangming Jin | 5 | 3 | 1.76 |