SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (24) , 25380-25388
Abstract
With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the speech features nodes within the graph and extracting the corresponding codebook by hierarchically and disentangledly minimizing 2D SE. Then, to address the issue of audio distortion, we propose a new quantization method. This method still adheres to the 2D SE minimization principle, adaptively selecting the most suitable token corresponding to the cluster for each incoming original speech node. Furthermore, we develop a Structural Entropy-based Speech Language Model (SESLM) that leverages SECodec. Experimental results demonstrate that SECodec performs comparably to EnCodec in speech reconstruction, and SESLM surpasses VALL-E in zero-shot text-to-speech tasks.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- Incremental measurement of structural entropy for dynamic graphsArtificial Intelligence, 2024
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Are Discrete Units Necessary for Spoken Language Modeling?IEEE Journal of Selected Topics in Signal Processing, 2022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- I2Transformer: Intra- and Inter-Relation Embedding Transformer for TV Show CaptioningIEEE Transactions on Image Processing, 2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Voice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter TrajectoryIEEE Transactions on Audio, Speech, and Language Processing, 2007
- A review of vector quantization techniquesIEEE Potentials, 2006