Ako
Authors:
- Raul Castro FernandezRFRaul Castro Fernandez
Open Access
- 5 October 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 84-97
Abstract
Distributed systems for the training of deep neural networks (DNNs) with large amounts of data have vastly improved the accuracy of machine learning models for image and speech recognition. DNN systems scale to large cluster deployments by having worker nodes train many model replicas in parallel; to ensure model convergence, parameter servers periodically synchronise the replicas. This raises the challenge of how to split resources between workers and parameter servers so that the cluster CPU and network resources are fully utilised without introducing bottlenecks. In practice, this requires manual tuning for each model configuration or hardware type. We describe Ako, a decentralised dataflow-based DNN system without parameter servers that is designed to saturate cluster resources. All nodes execute workers that fully use the CPU resources to update model replicas. To synchronise replicas as often as possible subject to the available network bandwidth, workers exchange partitioned gradient updates directly with each other. The number of partitions is chosen so that the used network bandwidth remains constant, independently of cluster size. Since workers eventually receive all gradient partitions after several rounds, convergence is unaffected. For the ImageNet benchmark on a 64-node cluster, Ako does not require any resource allocation decisions, yet converges faster than deployments with parameter serversKeywords
This publication has 22 references indexed in Scilit:
- Deep learningNature, 2015
- Optimistic Recovery for Iterative Dataflows in Action2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- MarianaProceedings of the VLDB Endowment, 2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Replicated data consistency explained through baseballCommunications of the ACM, 2013
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Distributed Asynchronous Incremental Subgradient Methods2001
- A bridging model for parallel computationCommunications of the ACM, 1990
- Learning representations by back-propagating errorsNature, 1986