dMel: Speech Tokenization made Simple
Preprint
Abstract
Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have investigated complicated speech tokenization methods to discretize continuous speech signals so that language modeling techniques can be applied to speech data. However, existing approaches either model semantic tokens, potentially losing acoustic information, or model acoustic tokens, risking the loss of semantic information. Having multiple token types also complicates the architecture and requires additional pretraining. Here we show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation (dMel), that performs better than other existing speech tokenization methods. Using a transformer decoder-only architecture for speech-text modeling, we comprehensively evaluate different speech tokenization methods on speech recognition (ASR), speech synthesis (TTS). Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio2023
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities2023
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition2019
- ESPnet: End-to-End Speech Processing Toolkit2018