DDNAS: Discretized Differentiable Neural Architecture Search for Text Classification
Open Access
- 3 October 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 14 (5) , 1-22
Abstract
Neural Architecture Search (NAS) has shown promising capability in learning text representation. However, existing text-based NAS neither performs a learnable fusion of neural operations to optimize the architecture nor encodes the latent hierarchical categorization behind text input. This article presents a novel NAS method, Discretized Differentiable Neural Architecture Search (DDNAS), for text representation learning and classification. With the continuous relaxation of architecture representation, DDNAS can use gradient descent to optimize the search. We also propose a novel discretization layer via mutual information maximization, which is imposed on every search node to model the latent hierarchical categorization in text representation. Extensive experiments conducted on eight diverse real datasets exhibit that DDNAS can consistently outperform the state-of-the-art NAS methods. While DDNAS relies on only three basic operations, i.e., convolution, pooling, and none, to be the candidates of NAS building blocks, its promising performance is noticeable and extensible to obtain further improvement by adding more different operations.Keywords
All Related Versions
Funding Information
- National Science and Technology Council (NSTC) of Taiwan (110-2221-E-006-136-MY3, 112-2628-E-006-012-MY3, 109-2118-M-006-010-MY2, 111-2221-E-006-001, 111-2634-F-002-022)
- Center for Data Science
- NCKU Miin Wu School of Computing
- Institute of Information Science (IIS), Academia Sinica
This publication has 10 references indexed in Scilit:
- The Surprising Performance of Simple Baselines for Misinformation Detection2021
- DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation2020
- Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction2020
- Feature selection for hierarchical classification via joint semantic and structural information of labelsKnowledge-Based Systems, 2020
- FENAS: Flexible and Expressive Neural Architecture Search2020
- A Formal Hierarchy of RNN Architectures2020
- Recent Trends in Deep Learning Based Natural Language Processing [Review Article]IEEE Computational Intelligence Magazine, 2018
- LSTM: A Search Space OdysseyIEEE Transactions on Neural Networks and Learning Systems, 2016
- A survey of hierarchical classification across different application domainsData Mining and Knowledge Discovery, 2010
- Long Short-Term MemoryNeural Computation, 1997