Can LLMs Truly Understand Relations? An Evaluation of Relation Extraction Capabilities

Author:
Open Access
Publisher website
Google Scholar
Abstract
Large language models (LLMs) are increasingly used for information extraction, and they often perform well when the target facts are stated plainly. In scientific papers, though, many relationships are only implied: the reader has to connect evidence across sentences, track context, and sometimes make several small inferences before a relation becomes clear. How reliably LLMs handle that kind of reading remains an open question. To study this, we propose an evaluation framework that tests relation extraction under progressively heavier reasoning demands. It relies on three prompt templates that grow more structured and more challenging, and we apply them to a curated perovskite solar cell dataset covering three core categories of scientific information—materials, processes, and performance. We then benchmark three representative models. The results show a consistent pattern. When relations are explicit, models extract them with high accuracy. As soon as the task requires deeper inference—especially for implicit or multi-hop relations that hinge on nuanced semantics—performance drops noticeably. The models also fail in different ways: some are comparatively good at maintaining coherent, step-by-step reasoning, while others are more dependable at avoiding hallucinated links or preserving type consistency. Even with these strengths, sizable gaps remain in coverage, reasoning depth, and overall completeness. Taken together, the findings clarify where current LLMs are genuinely useful for scientific information extraction and where they still fall short. They tend to be most reliable in structured extraction settings, while deeper semantic understanding remains the main bottleneck. By quantifying performance across prompt levels, the framework offers a practical yardstick for semantic comprehension and scientific reasoning in relation extraction, and it provides empirical support for downstream applications such as scientific knowledge graph construction.

This publication has 8 references indexed in Scilit: