Revisiting subword tokenization: A case study on affixal negation in large language models
Preprint
Abstract
In this work, we measure the impact of affixal negation on modern English large language models (LLMs). In affixal negation, the negated meaning is expressed through a negative morpheme, which is potentially challenging for LLMs as their tokenizers are often not morphologically plausible. We conduct extensive experiments using LLMs with different subword tokenization methods, which lead to several insights on the interaction between tokenization performance and negation sensitivity. Despite some interesting mismatches between tokenization accuracy and negation detection performance, we show that models can, on the whole, reliably recognize the meaning of affixal negation.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Inseq: An Interpretability Toolkit for Sequence Generation Models2023
- CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation2022
- The SIGMORPHON 2022 Shared Task on Morpheme Segmentation2022
- An Analysis of Negation in Natural Language Understanding Corpora2022
- The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation2021
- What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language ModelsTransactions of the Association for Computational Linguistics, 2020
- Byte Pair Encoding is Suboptimal for Language Model Pretraining2020
- An Analysis of Natural Language Inference Benchmarks through the Lens of Negation2020
- Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly2020
- BioLemmatizer: a lemmatization tool for morphological processing of biomedical textJournal of Biomedical Semantics, 2012