Enhancing Generalization in Chain of Thought Reasoning for Smaller Models
Preprint
Abstract
Chain-of-Thought (CoT) reasoning in smaller language models is a challenging natural language process problem yet highly desirable in many real-life applications. Existing CoT knowledge distillation methods often suffer from overly conservative memorization in smaller LLMs, leading to low generalization confidence. As fully preserving the CoT ability of teacher model is impossible, we hypothesize that adversarial CoT fine-tuning is crucial for developing smaller LLM with robust CoT generalization. To this end, we propose \textit{PRompt-Assisted Domain-Adversarial fine-tuning} (PRADA), a principled fine-tuning framework that integrates diverse CoT domains. Specifically, PRADA pioneers two CoT improvements in smaller LLM: (1) Recovering the domain-invariant feature insight which typically lost during distillation with domain adversarial fine-tuning; (2) Enhancing the domain adaptability of CoT prompt engineering by employing domain-adversarial approaches. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks. Moreover, our empirical findings reveal that the smaller LLM, when leveraging PRADA, aligns closely with domain knowledge, thereby improving the explainability of our approach.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Domain Adaptation via Prompt LearningIEEE Transactions on Neural Networks and Learning Systems, 2023
- Large Language Models Can Self-Improve2023
- Distilling Reasoning Capabilities into Smaller Language Models2023
- Large Language Models Are Reasoning Teachers2023
- Memorization and generalization in neural code intelligence modelsInformation and Software Technology, 2022
- Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- WARP: Word-level Adversarial ReProgramming2021