Multimodal deep learning using on-chip diffractive optics with in situ training capability
Top Cited Papers
Open Access
- 23 July 2024
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 15 (1) , 1-10
Abstract
Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm2), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low-power AI large models using photonic technology.Keywords
Funding Information
- National Natural Science Foundation of China (U21A20511)
This publication has 56 references indexed in Scilit:
- Large-scale Multi-modal Pre-trained Models: A Comprehensive SurveyMachine Intelligence Research, 2023
- Co-packaged optics (CPO): status, challenges, and solutionsFrontiers of Optoelectronics, 2023
- Si Microring Resonator Crossbar Array for On-Chip Inference and Training of the Optical Neural NetworkACS Photonics, 2022
- Neuromorphic Photonic Memory Devices Using Ultrafast, Non‐Volatile Phase‐Change MaterialsAdvanced Materials, 2022
- Circuit-level convergence of electronics and photonics: basic concepts and recent advancesFrontiers of Optoelectronics, 2022
- A small microring array that performs large complex-valued matrix-vector multiplicationFrontiers of Optoelectronics, 2022
- Analog Optical Computing for Artificial IntelligenceEngineering, 2022
- Self-Configuring and Reconfigurable Silicon Photonic Signal ProcessorACS Photonics, 2020
- The chips are down for Moore’s lawNature, 2016
- Single-chip microprocessor that communicates directly using lightNature, 2015