GLUECoS : An Evaluation Benchmark for Code-Switched NLP
Preprint
Abstract
Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition2019
- How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Language Informed Modeling of Code-Switched Text2018
- Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task2018
- Neural Architectures for Named Entity Recognition2016
- Part of Speech Tagging for Code Switched Data2016
- Multilingual Models for Compositional Distributed Semantics2014