DropDim: A Regularization Method for Transformer Networks
- 5 January 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 29 (10709908) , 474-478
Abstract
We introduce DropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is a key component of the transformer. In contrast to the general dropout method, which randomly drops neurons, DropDim drops part of the embedding dimensions. In this way, the semantic information can be completely discarded. Thus, the excessive co-adapting between different embedding dimensions can be broken, and the self-attention is forced to encode meaningful features with a certain number of embedding dimensions erased. Experiments on a wide range of tasks executed on the MUST-C English-Germany dataset show that DropDim can effectively improve model performance, reduce over-fitting, and show complementary effects with other regularization methods. When combined with label smoothing, the WER can be reduced from 19.1% to 15.1% on the ASR task, and the BLEU value can be increased from 26.90 to 28.38 on the MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase by 1.86 BLEU points compared to the strong baseline.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61673395, 62171470)
This publication has 17 references indexed in Scilit:
- Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss2020
- Adapting Transformer to End-to-End Spoken Language Translation2019
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition2019
- Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition2018
- Sequence-to-Sequence Models Can Directly Translate Foreign Speech2017
- Rethinking the Inception Architecture for Computer Vision2016
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Efficient object localization using Convolutional Networks2015
- BLEU2001