HyT-NAS: Hybrid Transformers Neural Architecture Search for Edge Devices
Open Access
- 21 September 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 41-45
Abstract
Vision Transformers have enabled recent attention-based Deep Learning (DL) architectures to achieve remarkable results in Computer Vision (CV) tasks. However, due to the extensive computational resources required, these architectures are rarely implemented on resource-constrained platforms. Current research investigates hybrid handcrafted convolution-based and attention-based models for CV tasks such as image classification and object detection. In this paper, we propose HyT-NAS, an efficient Hardware-aware Neural Architecture Search (HW-NAS) including hybrid architectures targeting vision tasks on tiny devices. HyT-NAS improves state-of-the-art HW-NAS by enriching the search space and enhancing the search strategy as well as the performance predictors. Our experiments show that HyT-NAS achieves a similar hypervolume with less than 5x training evaluations. Our resulting architecture outperforms MLPerf MobileNetVl by 6.3% accuracy improvement with 3.5x less number of parameters on Visual Wake Words.Keywords
All Related Versions
Funding Information
- CNRS
This publication has 11 references indexed in Scilit:
- DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object DetectionarXiv, 2022
- Rethinking Spatial Dimensions of Vision Transformers2021
- Accelerating Neural Architecture Search with Rank-Preserving Surrogate Models2021
- Hardware-Aware Neural Architecture Search: Survey and Taxonomy2021
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- SSD: Single Shot MultiBox Detector2016
- Adding Virtualization Capabilities to the Grid’5000 Testbed2013
- A medical robotic assistant for minimally invasive surgery2002
- A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer CodeTechnometrics, 2000
- Bayesian approach to global optimization and application to multiobjective and constrained problemsJournal of Optimization Theory and Applications, 1991