From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
Preprint
Abstract
The emergence of large language models (LLMs) has demonstrated that systems trained solely on text can acquire extensive world knowledge, develop reasoning capabilities, and internalize abstract semantic concepts--showcasing properties that can be associated with general intelligence. This raises an intriguing question: Do such concepts emerge in models trained on other modalities, such as speech? Furthermore, when models are trained jointly on multiple modalities: Do they develop a richer, more structured semantic understanding? To explore this, we analyze the conceptual structures learned by speech and textual models both individually and jointly. We employ Latent Concept Analysis, an unsupervised method for uncovering and interpreting latent representations in neural networks, to examine how semantic abstractions form across modalities. For reproducibility we made scripts and other resources available to the community.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations2025
- What Do Self-Supervised Speech Models Know About Words?Transactions of the Association for Computational Linguistics, 2024
- ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reportsEuropean Radiology, 2023
- Probing Self-supervised Speech Models for Phonetic and Phonemic Information: A Case Study in Aspiration2023
- What do end-to-end speech models learn about speaker, language and channel information? A layer-wise and neuron-level analysisComputer Speech & Language, 2023
- SLUE: New Benchmark Tasks For Spoken Language Understanding Evaluation on Natural Speech2022
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations2017
- Where do you know what you know? The representation of semantic knowledge in the human brainNature Reviews Neuroscience, 2007