Flexible silicon photonic architecture for accelerating distributed deep learning
- 9 January 2024
- journal article
- research article
- Published by Optica Publishing Group in Journal of Optical Communications and Networking
- Vol. 16 (2) , A157-A168
Abstract
The increasing size and complexity of deep learning (DL) models have led to the wide adoption of distributed training methods in datacenters (DCs) and high-performance computing (HPC) systems. However, communication among distributed computing units (CUs) has emerged as a major bottleneck in the training process. In this study, we propose Flex-SiPAC, a flexible silicon photonic accelerated compute cluster designed to accelerate multi-tenant distributed DL training workloads. Flex-SiPAC takes a co-design approach that combines a silicon photonic hardware platform with a tailored collective algorithm, optimized to leverage the unique physical properties of the architecture. The hardware platform integrates a novel wavelength-reconfigurable transceiver design and a micro-resonator-based wavelength-reconfigurable switch, enabling the system to achieve flexible bandwidth steering in the wavelength domain. The collective algorithm is designed to support reconfigurable topologies, enabling efficient all-reduce communications that are commonly used in DL training. The feasibility of the Flex-SiPAC architecture is demonstrated through two testbed experiments. First, an optical testbed experiment demonstrates the flexible routing of wavelengths by shuffling an array of input wavelengths using a custom-designed spatial-wavelength selective switch. Second, a four-GPU testbed running two DL workloads shows a 23% improvement in job completion time compared to a similarly sized leaf-spine topology. We further evaluate Flex-SiPAC using large-scale simulations, which show that Flex-SiPAC is able to reduce the communication time by 26% to 29% compared to state-of-the-art compute clusters under representative collective operations.Keywords
Funding Information
- Advanced Research Projects Agency - Energy (DE-AR00000843)
- National Security Agency (FA8075-14-D-0002-0007, TAT 15-1158)
- Center for Ubiquitous Connectivity
- Semiconductor Research Corporation
- Defense Advanced Research Projects Agency
This publication has 38 references indexed in Scilit:
- RDON: a rack-scale disaggregated data center network based on a distributed fast optical switchJournal of Optical Communications and Networking, 2020
- PULSE: Optical Circuit Switched Data Center Architecture Operating at Nanosecond TimescalesJournal of Lightwave Technology, 2020
- Ultra-dense optical data transmission over standard fibre with a single chip sourceNature Communications, 2020
- Flexspander: augmenting expander networks in high-performance systems with optical bandwidth steeringJournal of Optical Communications and Networking, 2020
- Scalable Microring-Based Silicon Clos Switch Fabric With Switch-and-Select StagesIEEE Journal of Selected Topics in Quantum Electronics, 2019
- A 128 Gb/s PAM4 Silicon Microring Modulator With Integrated Thermo-Optic Resonance TuningJournal of Lightwave Technology, 2018
- Flexfly: Enabling a Reconfigurable Dragonfly through Silicon Photonics2016
- BCube2009
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- New methods to color the vertices of a graphCommunications of the ACM, 1979