Integrating Eye Tracking With Grouped Fusion Networks for Semantic Segmentation on Mammogram Images
- 27 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 44 (2) , 868-879
Abstract
Medical image segmentation has seen great progress in recent years, largely due to the development of deep neural networks. However, unlike in computer vision, high-quality clinical data is relatively scarce, and the annotation process is often a burden for clinicians. As a result, the scarcity of medical data limits the performance of existing medical image segmentation models. In this paper, we propose a novel framework that integrates eye tracking information from experienced radiologists during the screening process to improve the performance of deep neural networks with limited data. Our approach, a grouped hierarchical network, guides the network to learn from its faults by using gaze information as weak supervision. We demonstrate the effectiveness of our framework on mammogram images, particularly for handling segmentation classes with large scale differences.We evaluate the impact of gaze information on medical image segmentation tasks and show that our method achieves better segmentation performance compared to state-of-the-art models. A robustness study is conducted to investigate the influence of distraction or inaccuracies in gaze collection. We also develop a convenient system for collecting gaze data without interrupting the normal clinical workflow. Our work offers novel insights into the potential benefits of integrating gaze information into medical image segmentation tasks.Keywords
Funding Information
- China Ministry of Science and Technology, STI 2030-Major Projects (2022ZD0213100,2022ZD0209000)
- National Natural Science Foundation of China (U23A20295, 62131015, 82394432)
- Shanghai Municipal Central Guided Local Science and Technology Development Fund (YDZX20233100001001)
- the Key R&D Program of Guangdong Province (2023B0303040001, 2021B0101420006)
This publication has 36 references indexed in Scilit:
- Eye Tracking for Deep Learning Segmentation Using Convolutional Neural NetworksJournal of Digital Imaging, 2019
- A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learningMedical Image Analysis, 2019
- UNet++: A Nested U-Net Architecture for Medical Image Segmentation2018
- Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Looking in the Same Manner but Seeing it Differently: Bottom‐up and Expertise Effects in RadiologyApplied Cognitive Psychology, 2012
- INbreastAcademic Radiology, 2012
- Iterative multi-atlas-based multi-image segmentation with tree-based registrationNeuroImage, 2012
- Using Gaze-tracking Data and Mixture Distribution Analysis to Support a Holistic Model for the Detection of Cancers on MammogramsAcademic Radiology, 2008
- Some Aspects of the Selective Process in the Functional Visual FieldErgonomics, 1970