Knowledge-Decoupled Functionally Invariant Path With Synthetic Personal Data for Personalized ASR
- 14 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 32 (10709908) , 4024-4028
Abstract
Fine-tuning generic ASR models with large-scale synthetic personal data can enhance the personalization of ASR models, but it introduces challenges in adapting to synthetic personal data without forgetting real knowledge, and in adapting to personal data without forgetting generic knowledge. Considering that the functionally invariant path (FIP) framework enables model adaptation while preserving prior knowledge, in this letter, we introduce FIP into synthetic-data-augmented personalized ASR models. However, the model still struggles to balance the learning of synthetic, personalized, and generic knowledge when applying FIP to train the model on all three types of data simultaneously. To decouple this learning process and further address the above two challenges, we integrate a gated parameter-isolation strategy into FIP and propose a knowledge-decoupled functionally invariant path (KDFIP) framework, which stores generic and personalized knowledge in separate modules and applies FIP to them sequentially. Specifically, KDFIP adapts the personalized module to synthetic and real personal data and the generic module to generic data. Both modules are updated along personalization-invariant paths, and their outputs are dynamically fused through a gating mechanism. With augmented synthetic data, KDFIP achieves a 29.38% relative character error rate reduction on target speakers and maintains comparable generalization performance to the unadapted ASR baseline.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62376071)
This publication has 31 references indexed in Scilit:
- Engineering flexible machine learning systems by traversing functionally invariant pathsNature Machine Intelligence, 2024
- Text-Only Unsupervised Domain Adaptation for Neural Transducer-Based ASR Personalization Using Synthesized Data2024
- Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding2023
- Unsupervised Speaker Adaptation Using Attention-Based Speaker Memory for End-to-End ASR2020
- Using Personalized Speech Synthesis and Neural Language Generator for Rapid Speaker Adaptation2020
- Speaker-Aware Speech-Transformer2019
- X-Vectors: Robust DNN Embeddings for Speaker Recognition2018
- Unsupervised speaker adaptation of batch normalized acoustic models for robust ASR2017
- Improving DNN speaker independence with I-vector inputs2014
- Speaker adaptation of neural network acoustic models using i-vectors2013