NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks
Preprint
Abstract
Siamese networks are one of the most trending methods to achieve self-supervised visual representation learning (SSL). Since hand labeling is costly, SSL can play a crucial part by allowing deep learning to train on large unlabeled datasets. Meanwhile, Neural Architecture Search (NAS) is becoming increasingly important as a technique to discover novel deep learning architectures. However, early NAS methods based on reinforcement learning or evolutionary algorithms suffered from ludicrous computational and memory costs. In contrast, differentiable NAS, a gradient-based approach, has the advantage of being much more efficient and has thus retained most of the attention in the past few years. In this article, we present NASiam, a novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor (encoder/predictor pair) architectures inside siamese-networks-based contrastive learning frameworks (e.g., SimCLR, SimSiam, and MoCo) while preserving the simplicity of previous baselines. We crafted a search space designed explicitly for multilayer perceptrons, inside which we explored several alternatives to the standard ReLU activation function. We show that these new architectures allow ResNet backbone convolutional models to learn strong representations efficiently. NASiam reaches competitive performance in both small-scale (i.e., CIFAR-10/CIFAR-100) and large-scale (i.e., ImageNet) image classification datasets while costing only a few GPU hours. We discuss the composition of the NAS-discovered architectures and emit hypotheses on why they manage to prevent collapsing behavior. Our code is available at https://github.com/aheuillet/NASiam.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- D-DARTS: Distributed Differentiable Architecture SearchPattern Recognition Letters, 2023
- Towards Self-Supervised and Weight-preserving Neural Architecture Search2023
- CSNAS: Contrastive Self-Supervised Learning Neural Architecture Search Via Sequential Model-Based OptimizationIEEE Transactions on Artificial Intelligence, 2021
- Progressive DARTS: Bridging the Optimization Gap for NAS in the WildInternational Journal of Computer Vision, 2020
- Microsoft COCO: Common Objects in Context2014
- SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK1994
- Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition1990