Eliciting Better Multilingual Structured Reasoning from LLMs through Code
Preprint
Abstract
Development of large language models (LLM) have shown progress on reasoning, though studies have been limited to English or simple reasoning tasks. We thus introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multi-lingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus showing our techniques maintain general-purpose abilities.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- CodeKGC: Code Language Model for Generative Knowledge Graph ConstructionACM Transactions on Asian and Low-Resource Language Information Processing, 2024
- MEGA: Multilingual Evaluation of Generative AI2023
- Crosslingual Generalization through Multitask Finetuning2023
- Language Models of Code are Few-Shot Commonsense Learners2022