DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression
Authors:
Preprint
Abstract
Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. With many open-source LLMs available, selecting the best model for fine-tuning downstream tasks is challenging, primarily focusing on how to quickly identify the optimal LLM. We introduce a Data and Model Compression Framework (DaMoC) that addresses this challenge by: 1) Data Level: A systematic categorization of data filtering methodologies for LLMs is first established, classifying them into three distinct paradigms: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. Further, we enhance the density of key tokens in the text achieving token compression. Subsequently, we use an LLM to iterative rewrite the text to optimize its expression. 2) Model Level: We use layer similarity scores to assess each layer's importance and remove those with lower importance. Then, we introduce a sparse merging paradigm to preserve as much of the original model's capability as possible. Extensive experiments on four datasets, medical Q&A, financial Q&A, general Q&A, and reading comprehension, show that we can select the optimal LLM while saving approximately 20-fold in training time.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- SWIFT: A Scalable Lightweight Infrastructure for Fine-TuningProceedings of the AAAI Conference on Artificial Intelligence, 2025
- A Survey on Model Compression for Large Language ModelsTransactions of the Association for Computational Linguistics, 2024
- Fluctuation-Based Adaptive Structured Pruning for Large Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation2024
- LaCo: Large Language Model Pruning via Layer Collapse2024
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning2024
- Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model2024
- LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression2024
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models2023
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016