Exploring Multimodal Approaches for Alzheimer's Disease Detection Using Patient Speech Transcript and Audio Data
Preprint
Abstract
Alzheimer's disease (AD) is a common form of dementia that severely impacts patient health. As AD impairs the patient's language understanding and expression ability, the speech of AD patients can serve as an indicator of this disease. This study investigates various methods for detecting AD using patients' speech and transcripts data from the DementiaBank Pitt database. The proposed approach involves pre-trained language models and Graph Neural Network (GNN) that constructs a graph from the speech transcript, and extracts features using GNN for AD detection. Data augmentation techniques, including synonym replacement, GPT-based augmenter, and so on, were used to address the small dataset size. Audio data was also introduced, and WavLM model was used to extract audio features. These features were then fused with text features using various methods. Finally, a contrastive learning approach was attempted by converting speech transcripts back to audio and using it for contrastive learning with the original audio. We conducted intensive experiments and analysis on the above methods. Our findings shed light on the challenges and potential solutions in AD detection using speech and audio data.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- AugGPT: Leveraging ChatGPT for Text Data AugmentationIEEE Transactions on Big Data, 2025
- Graph Neural Networks for Natural Language Processing: A SurveyFoundations and Trends in Machine Learning, 2023
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- An automatic Alzheimer’s disease classifier based on spontaneous spoken EnglishComputer Speech & Language, 2022
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing2022
- Transformer-based deep neural network language models for Alzheimer’s disease risk assessment from targeted speechBMC Medical Informatics and Decision Making, 2021
- Contextualized Perturbation for Textual Adversarial Attack2021
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- Deep & Cross Network for Ad Click Predictions2017
- A Primer on Neural Network Models for Natural Language ProcessingJournal of Artificial Intelligence Research, 2016