Benchmarking Machine Translation on Chinese Social Media Texts
Authors:
Preprint
Abstract
The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese social media, poses significant challenges for Machine Translation (MT) benchmarking. Specifically, we identify two primary obstacles: (1) data scarcity, as high-quality parallel data requires bilingual annotators familiar with platform-specific slang, and stylistic cues in both languages; and (2) metric limitations, where traditional evaluators like COMET often fail to capture stylistic fidelity and nonstandard expressions. To bridge these gaps, we introduce CSM-MTBench, a benchmark covering five Chinese-foreign language directions and consisting of two expert-curated subsets: Fun Posts, featuring context-rich, slang- and neologism-heavy content, and Social Snippets, emphasizing concise, emotion- and style- driven expressions. Furthermore, we propose tailored evaluation approaches for each subset: measuring the translation success rate of slang and neologisms in Fun Posts, while assessing tone and style preservation in Social Snippets via a hybrid of embedding-based metrics and LLM-as-a-judge. Experiments on over 20 models reveal substantial variation in how current MT systems handle semantic fidelity and informal, social-media-specific stylistic cues. CSM-MTBench thus serves as a rigorous testbed for advancing MT systems capable of mastering real-world Chinese social media texts.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- A survey on LLM-as-a-judgeThe Innovation, 2026
- xcomet : Transparent Machine Translation Evaluation through Fine-grained Error DetectionTransactions of the Association for Computational Linguistics, 2024
- Translate Meanings, Not Just Words: IdiomKB’s Role in Optimizing Idiomatic Translation with Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Document-Level Machine Translation with Large Language Models2023
- Prompting PaLM for Translation: Assessing Strategies and Performance2023
- WYWEB: A NLP Evaluation Benchmark For Classical Chinese2023
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research2021
- COMET: A Neural Framework for MT Evaluation2020
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017