Chapter 1: Introduction1.1. Background
1.2. Distributed machine learning
1.3. Gradient optimization
1.4. Challenges
Chapter 2: The preliminaries
2.1. Overview
2.2. Parallel strategy
2.3. Gradient compression
2.4. Synchronization protocol
Chapter 3: Parallel strategy
1.1. Background and problem
1.2. Data parallelism
1.3. Model parallelism
1.4. Hybrid parallelism
3.5. Benchmark
3.6. Summary
Chapter 4: Gradient compression 4.1. Background and problem
4.2. Lossless gradient compression
4.3. Lossy gradient compression
4.4. Sparse gradient compression
4.5. Benchmark
4.6. Summary
Chapter 5: Synchronization protocol
5.1. Background and problem
5.2. Bulk synchronous protocol
5.3. Asynchronous protocol 5.4. Stale synchronous protocol
5.5. Benchmark
5.6. Summary Chapter 6: Conclusion
6.1. Summary of the book
6.2. Future work
About the Author: Jiawei Jiang obtained his PhD from Peking University 2018, advised by Prof. Bin Cui. His research interests include distributed machine learning, gradient optimization and automatic machine learning. He has served as a program committee member or reviewer for various international events, including SIGMOD, VLDB, ICDE, KDD, AAAI and TKDE. He was awarded the CCF Outstanding Doctoral Dissertation Award (2019) and ACM China Doctoral Dissertation Award (2018).
Bin Cui is a Professor at the School of EECS and Director of the Institute of Network Computing and Information Systems, at Peking University. His research interests include database system architectures, query and index techniques, and big data management and mining. He has published over 200 refereed papers at international conferences and in journals. Dr. Cui has served on the technical program committee of various international conferences, including SIGMOD, VLDB, ICDE and KDD, and as Vice PC Chair of ICDE 2011, Demo Co-Chair of ICDE 2014, Area Chair of VLDB 2014, PC Co-Chair of APWeb 2015 and WAIM 2016. He is currently a member of the trustee board of VLDB Endowment, is on the editorial board of the VLDB Journal, Distributed and Parallel Databases Journal, and Information Systems, and was formerly an associate editor of IEEE Transactions on Knowledge and Data Engineering (TKDE, 2009-2013). He was selected for a Microsoft Young Professorship award (MSRA 2008), CCF Young Scientist award (2009), Second Prize of Natural Science Award of MOE China (2014), and appointed a Cheung Kong distinguished Professor by the MOE in 2016.
