Locating X-Ray Coronary Angiogram Keyframes via Long Short-Term Spatiotemporal Attention With Image-to-Patch Contrastive Learning
- 16 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 43 (1) , 51-63
Abstract
Locating the start, apex and end keyframes of moving contrast agents for keyframe counting in X-ray coronary angiography (XCA) is very important for the diagnosis and treatment of cardiovascular diseases. To locate these keyframes from the class-imbalanced and boundary-agnostic foreground vessel actions that overlap complex backgrounds, we propose long short-term spatiotemporal attention by integrating a convolutional long short-term memory (CLSTM) network into a multiscale Transformer to learn the segment-and sequence-level dependencies in the consecutive-frame-based deep features. Image-to-patch contrastive learning is further embedded between the CLSTM-based long-term spatiotemporal attention and Transformer-based short-term attention modules. The imagewise contrastive module reuses the long-term attention to contrast image-level foreground/background of XCA sequence, while patchwise contrastive projection selects the random patches of backgrounds as convolution kernels to project foreground/background frames into different latent spaces. A new XCA video dataset is collected to evaluate the proposed method. The experimental results show that the proposed method achieves a mAP (mean average precision) of 72.45% and a F-score of 0.8296, considerably outperforming the state-of-the-art methods. The source code and dataset are available at https://github.com/Binjie-Qin/STA-IPCon.Keywords
All Related Versions
Funding Information
- Science and Technology Commission of Shanghai Municipality (19dz1200500, 19411951507)
- National Natural Science Foundation of China (61271320, 82070477)
- Shanghai Science and Technology Committee (22XD1421800)
- Shanghai “Rising Stars of Medical Talent” Youth Development Program “Outstanding Youth Medical Talents” (SHWSRS (2021) 099)
- New Interdisciplinary Research Project of Shanghai Municipal Health Commission (2022JC022)
- Shanghai Jiao Tong University Medical-Engineering Cross Research Fund (ZH2018ZDA19, YG2021QN122, YG2021QN99)
This publication has 72 references indexed in Scilit:
- Hierarchical Recurrent Neural Network for Video Summarization2017
- Automatic online layer separation for vessel enhancement in X-ray angiograms for percutaneous coronary interventionsMedical Image Analysis, 2017
- Unsupervised Video Summarization with Adversarial LSTM Networks2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Extracting contrast-filled vessels in X-ray angiography by graduated RPCA with motion coherency constraintPattern Recognition, 2017
- Motion-State-Adaptive Video Summarization via Spatiotemporal AnalysisIEEE Transactions on Circuits and Systems for Video Technology, 2016
- Frame counting improves the assessment of post-reperfusion microvascular patency by TIMI myocardial perfusion grade: Evidence from cardiac magnetic resonance imagingInternational Journal of Cardiology, 2016
- A flexible patch based approach for combined denoising and contrast enhancement of digital X-ray imagesMedical Image Analysis, 2015
- RPCA-KFE: Key Frame Extraction for Video Using Robust Principal Component AnalysisIEEE Transactions on Image Processing, 2015
- TIMI myocardial perfusion frame count: A new method to assess myocardial perfusion and its predictive value for short‐term prognosisCatheterization and Cardiovascular Interventions, 2010