DNA-T: Deformable Neighborhood Attention Transformer for Irregular Medical Time Series
- 30 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 28 (7) , 4224-4237
Abstract
The real-world Electronic Health Records (EHRs) present irregularities due to changes in the patient's health status, resulting in various time intervals between observations and different physiological variables examined at each observation point. There have been recent applications of Transformer-based models in the field of irregular time series. However, the full attention mechanism in Transformer overly focuses on distant information, ignoring the short-term correlations of the condition. Thereby, the model is not able to capture localized changes or short-term fluctuations in patients' conditions. Therefore, we propose a novel end-to-end Deformable Neighborhood Attention Transformer (DNA-T) for irregular medical time series. The DNA-T captures local features by dynamically adjusting the receptive field of attention and aggregating relevant deformable neighborhoods in irregular time series. Specifically, we design a Deformable Neighborhood Attention (DNA) module that enables the network to attend to relevant neighborhoods by drifting the receiving field of neighborhood attention. The DNA enhances the model's sensitivity to local information and representation of local features, thereby capturing the correlation of localized changes in patients' conditions. We conduct extensive experiments to validate the effectiveness of DNA-T, outperforming existing state-of-the-art methods in predicting the mortality risk of patients. Moreover, we visualize an example to validate the effectiveness of the proposed DNA. Our code is available at https://github.com/Nekumiya-x/DNA-T .Keywords
Funding Information
- National Natural Science Foundation of China (62102098)
- Science and Technology Planning Project of Guangzhou (202201010266)
- National Natural Science Foundation of China (62302413)
- Natural Science Foundation of Guangdong Province (2024A1515010186)
- Regional Joint Fund Project of Basic and Applied Basic Research Foundation of Guangdong Province (2022A1515140096)
This publication has 29 references indexed in Scilit:
- Neighborhood Attention Transformer2023
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Scaling Local Self-Attention for Parameter Efficient Visual Backbones2021
- Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models2020
- Deep RNN Framework for Visual Sequential Applications2019
- Recurrent Neural Networks for Multivariate Time Series with Missing ValuesScientific Reports, 2018
- Personalized disease prediction using a CNN-based similarity learning method2017
- Deformable Convolutional Networks2017
- $\mathtt {Deepr}$: A Convolutional Net for Medical RecordsIEEE Journal of Biomedical and Health Informatics, 2016
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016