Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding
- 4 June 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2972-2985
Abstract
Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance, limiting their use and performance in diagnostic tasks. To bridge this gap, we introduce AcuLa (Audio–Clinical Understanding via Language Alignment), a lightweight post-training framework that instills semantic understanding into any audio encoder by aligning it with a medical language model, which acts as a “semantic teacher.” To enable alignment at scale, we construct a large-scale dataset by leveraging off-the-shelf large language models to translate the rich, structured metadata accompanying existing audio recordings into coherent clinical reports. Our alignment strategy combines a representation-level contrastive objective with a self-supervised modeling, ensuring that the model learns clinical semantics while preserving fine-grained temporal cues. AcuLa achieves state-of-the-art results across 18 diverse cardio-respiratory tasks from 10 different datasets, improving the mean AUROC on classification benchmarks from 0.68 to 0.79 and, on the most challenging COVID-19 cough detection task, boosting the AUROC from 0.55 to 0.89. Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools, establishing a new paradigm for infusing clinical-language semantics into audio representations for audio-based health monitoring.Keywords
All Related Versions
Funding Information
- NWO AiNed Fellowship
- Google Cloud Research Credits Program
- Gemini Academic Program
This publication has 15 references indexed in Scilit:
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature, 2025
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models2025
- CLAP Learning Audio Concepts from Natural Language Supervision2023
- ImageBind One Embedding Space to Bind Them All2023
- Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens2023
- The CirCor DigiScope Dataset: From Murmur Detection to Murmur ClassificationIEEE Journal of Biomedical and Health Informatics, 2022
- AugLy: Data Augmentations for Adversarial Robustness2022
- Audioclip: Extending Clip to Image, Text and Audio2022
- Contrastive Learning of General-Purpose Audio Representations2021
- An open access database for the evaluation of respiratory sound classification algorithmsPhysiological Measurement, 2019