Abstract
The parameter server (PS) paradigm has achieved great success in deploying large-scale distributed Deep Learning (DL) systems. However, the implicit assumption of homogeneous clusters in these systems does not hold in many real-world cases. Also, existing heterogeneity-aware approaches mainly prioritize the contribution of fast workers and reduce the involvement of slow workers, resulting in the limitations of workload imbalance and computation inefficiency. We reveal that grouping workers into communities, an abstraction proposed by us, and handling parameter synchronization in community level can conquer these limitations and accelerate training convergence progress. The inspiration of community comes from our exploration of prior knowledge about the similarity between workers, which is often neglected by previous work. These observations motivate us to propose a new synchronization mechanism named Community-aware Synchronous Parallel (CASP), which uses the Asynchronous Advantage Actor-Critic (A3C) to intelligently determine community configuration. The whole idea has been implemented in a system called Petrel that achieves a good balance between convergence efficiency and communication overhead. Evaluation under various benchmarks demonstrates the superiority of Petrel. Specifically, Petrel accelerates the training convergence speed by up to 1.87 times faster and reduces communication traffic by up to 26.85%, on average, over the non-community synchronization mechanisms.
Funding Information
  • Hong Kong RGC Research Impact Fund (R5060-19, R5034-18)
  • General Research Fund (152221/19E)
  • Collaborative Research Fund (C5026-18G)
  • National Natural Science Foundation of China (61872310)
  • China Postdoctoral Science Foundation (2019M661709)
  • Shanghai Trusted Industrial Control Platform