A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists
Authors:
- Adrian MirzaAMAdrian Mirza
- Nawaf AlamparaNANawaf Alampara
- Sreekanth KunchapuSKSreekanth Kunchapu
- Martiño Ríos-GarcíaMRMartiño Ríos-García
- Benedict EmoekabuBEBenedict Emoekabu
- Aswanth KrishnanAKAswanth Krishnan
- Tanya GuptaTGTanya Gupta
- Mara Schilling-WilhelmiMSMara Schilling-Wilhelmi
- Macjonathan OkerekeMOMacjonathan Okereke
- Anagha AneeshAAAnagha Aneesh
- Mehrdad AsgariMAMehrdad Asgari
- Juliane EberhardtJEJuliane Eberhardt
Top Cited Papers
Open Access
- 20 May 2025
- journal article
- Published by Springer Nature in Nature Chemistry
- Vol. 17 (7) , 1027-1034
Abstract
Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question–answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs’ impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.Keywords
Funding Information
- Carl-Zeiss-Stiftung
- Helmholtz Association
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (225147)
- Fulbright Association
- EC | Horizon 2020 Framework Programme (101106377)
- Ministry of Economy and Competitiveness | Agencia Estatal de Investigación (CNS2022-135474)
- Deutsche Forschungsgemeinschaft (497115849, 497115849)
This publication has 52 references indexed in Scilit:
- Comprehensive evaluation of molecule property prediction with ChatGPTMethods, 2024
- Autonomous chemical research with large language modelsNature, 2023
- Fine-tuning GPT-3 for machine learning electronic and functional properties of organic moleculesChemical Science, 2023
- Extracting medicinal chemistry intuition via preference machine learningNature Communications, 2023
- 14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathonDigital Discovery, 2023
- The future of chemistry is languageNature Reviews Chemistry, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- A teachable moment for dual-useNature Machine Intelligence, 2022
- Dual use of artificial-intelligence-powered drug discoveryNature Machine Intelligence, 2022
- MoleculeNet: a benchmark for molecular machine learningChemical Science, 2017