The Price of Format: Diversity Collapse in LLMs
Preprint
Abstract
Instruction-tuned large language models (LLMs) employ structured templates, such as role markers and special tokens, to enforce format consistency during inference. However, we identify a critical limitation of such formatting: it induces a phenomenon we term diversity collapse, where the model generates semantically similar outputs for open-ended inputs, undermining creativity and variability. We systematically evaluate this effect across tasks like story completion and free-form generation, finding that (1) diversity collapse persists even under high-temperature sampling, and (2) structural tokens in templates significantly constrain the model's output space. To contextualize these findings, we fine-tune the same model using a range of structured prompts and then evaluate them across three axes: downstream task performance, alignment behavior, and output diversity. Our analysis shows that format consistency between fine-tuning and inference is crucial for structure-sensitive tasks (e.g., GSM8K, IFEval), but has marginal influence on knowledge-heavy tasks (e.g., MMLU, WebQuestions). In contrast, output diversity is primarily governed by the presence or absence of structural tokens, with minimal formatting yielding the most diverse outputs. These findings reveal that current prompting conventions, while beneficial for alignment, may inadvertently suppress output diversity, underscoring the need for diversity-aware prompt design and instruction tuning.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- RewardBench: Evaluating Reward Models for Language Modeling2025
- POSTER: Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications2024
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting2023
- Measuring and Improving Semantic Diversity of Dialogue Generation2022
- Evaluating the Evaluation of Diversity in Natural Language Generation2021
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- ELI5: Long Form Question Answering2019
- Hierarchical Neural Story Generation2018
- LSDSem 2017 Shared Task: The Story Cloze Test2017