MUKA: Multi Kernel Audio Adaptation Of Audio-Language Models
Preprint
Abstract
Multimodal foundation models have demonstrated impressive generalization capabilities, yet efficiently adapting them to new tasks in a few-shot setting remains a critical challenge. In this work, we investigate the few-shot adaptation of Large Audio-Language Models (ALMs) through both training-based and training-free approaches. We introduce MUKA, a multi-kernel adaptation framework that combines the fine-grained, context-dependent representations of instruction-tuning based models like Pengi with the global semantic representations of contrastive pretraining methods like CLAP. By constructing a product kernel that aligns local similarity with global semantics, MUKA enhances representational power while preserving the theoretical guarantees of kernel methods and avoiding additional training. Extensive experiments across 11 diverse audio datasets demonstrate that MUKA achieves state-of-the-art performance among training-free methods and even surpasses training-based adapters in several scenarios, offering a compelling balance between adaptability and efficiency.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models2025
- Natural Language Supervision For General-Purpose Audio Representations2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- CLAP Learning Audio Concepts from Natural Language Supervision2023
- MaPLe: Multi-modal Prompt Learning2023
- Reproducible Scaling Laws for Contrastive Language-Image Learning2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Audioclip: Extending Clip to Image, Text and Audio2022
- HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection2022
- A study of instrument-wise onset detection in Beijing Opera percussion ensembles2014