CMNN: Coupled Modular Neural Network
Open Access
- 30 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 93871-93891
Abstract
In this paper, we propose a multi-branch neural network architecture named Coupled Modular Neural Network (CMNN). A CMNN is a network consisting of $\beta $ closely coupled sub-networks, where $\beta $ is termed as the branching factor in this paper. We call the whole network a super-graph and each sub-network a sub-graph. Each sub-graph is a stand-alone neural network and shares a common block with other sub-graphs. To effectively leverage the super-graph we propose a simple but easy-to-implement Round-Robin-based learning algorithm. Each training iteration contains two phases. In the first phase, we choose a sub-graph in a Round-Robin fashion and train it using knowledge of the super-graph (distillation). In the second phase, we fine-tune the super-graph based on the updated sub-graphs. This algorithm produces a different copy of the super-graph at each iteration which acts as an improved teacher network for the sub-graph; and a different copy of one of the sub-graphs which functions as a new building block for the super-graph. To validate and test CMNN and the proposed algorithm, we conduct experiments on CIFAR-10, CIFAR-100, Tiny ImageNet and a private On-Road-Risk (ORR) datasets. Empirical results on all these four datasets indicate that we not only obtain a strong sub-graph network, the learning framework can also produce strong ensemble performance which substantiates the diversity introduced throughout the learning framework.
Keywords
This publication has 34 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Model compression2006
- Ensemble selection from libraries of models2004
- Stochastic gradient boostingComputational Statistics & Data Analysis, 2002
- A Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingJournal of Computer and System Sciences, 1997
- Bagging predictorsMachine Learning, 1996
- The Backpropagation Algorithm1996
- Modularity and scaling in large phonemic neural networksIEEE Transactions on Acoustics, Speech, and Signal Processing, 1989
- Neocognitron: A hierarchical neural network capable of visual pattern recognitionNeural Networks, 1988