Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
Preprint
Abstract
Ancient Chinese text processing presents unique challenges for large language models (LLMs) due to its distinct linguistic features, complex structural constraints, and rich cultural context. While existing benchmarks have primarily focused on evaluating comprehension through multiple-choice questions, there remains a critical gap in assessing models' generative capabilities in classical Chinese. We introduce Fùxì, a comprehensive benchmark that evaluates both understanding and generation capabilities across 21 diverse tasks. Our benchmark distinguishes itself through three key contributions: (1) balanced coverage of both comprehension and generation tasks, including novel tasks like poetry composition and couplet completion, (2) specialized evaluation metrics designed specifically for classical Chinese text generation, combining rule-based verification with fine-tuned LLM evaluators, and (3) a systematic assessment framework that considers both linguistic accuracy and cultural authenticity. Through extensive evaluation of state-of-the-art LLMs, we reveal significant performance gaps between understanding and generation tasks, with models achieving promising results in comprehension but struggling considerably in generation tasks, particularly those requiring deep cultural knowledge and adherence to classical formats. Our findings highlight the current limitations in ancient Chinese text processing and provide insights for future model development. The benchmark, evaluation toolkit, and baseline results are publicly available to facilitate research in this domain.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Evaluation of Ancient Chinese Natural Language Understanding in Large Language Models Based on ACHNLU2024
- WITHDRAWN: Evaluating the generation capabilities of large Chinese language modelsAI Open, 2024
- CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM2024
- Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluation2023
- CLEVA: Chinese Language Models EVAluation Platform2023
- Yu Sheng: Human-in-Loop Classical Chinese Poetry Generation System2023
- WYWEB: A NLP Evaluation Benchmark For Classical Chinese2023
- C-CLUE: A Benchmark of Classical Chinese Based on a Crowdsourcing System for Knowledge Graph Construction2021
- A Sentiment and Style Controllable Approach for Chinese Poetry Generation2021
- Poetic Expression Through Scenery: Sentimental Chinese Classical Poetry Generation from Images2021