MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages
Authors:
- Weerayut BuaphetWBWeerayut Buaphet
- Thanh-Nhi NguyenTNThanh-Nhi Nguyen
- Risa KondoRKRisa Kondo
- Tomoyuki KajiwaraTKTomoyuki Kajiwara
- Yumin KimYKYumin Kim
- Jimin LeeJLJimin Lee
- Hwanhee LeeHLHwanhee Lee
- Holy LoveniaHLHoly Lovenia
- Peerat LimkonchotiwatPLPeerat Limkonchotiwat
- Sarana NutanongSNSarana Nutanong
- Rob Van der GootRGRob Van der Goot
Open Access
- 9 July 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Asian and Low-Resource Language Information Processing
- Vol. 25 (7) , 1-14
Abstract
Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers five Asian languages from different language families in four different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine TranslationTransactions of the Association for Computational Linguistics, 2024
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte ModelsTransactions of the Association for Computational Linguistics, 2022
- Statistical Machine Translation Approach for Lexical Normalization on Indonesian Text2020
- DaN+: Danish Nested Named Entities and Lexical Normalization2020
- BERTweet: A pre-trained language model for English Tweets2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- Self-Attention with Cross-Lingual Position Representation2020
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks2020
- How to tag non-standard language: Normalisation versus domain adaptation for Slovene historical and user-generated textsNatural Language Engineering, 2019
- Multimodular Text Normalization of Dutch User-Generated ContentACM Transactions on Intelligent Systems and Technology, 2016