RARE: Retrieval-Augmented Reasoning Modeling

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Domain-specific intelligence demands specialized knowledge and sophisticated reasoning for problem-solving, posing significant challenges for large language models (LLMs) that struggle with knowledge hallucination and inadequate reasoning capabilities. For the efficient discovery and modeling of reasoning patterns, we propose Retrieval-Augmented Reasoning Modeling (RARE), a novel paradigm to extract and model reasoning patterns from data, represented as salient and learnable tokens. Specifically, RARE externalizes domain knowledge to retrievable sources and internalizes reasoning patterns in LLMs. By injecting retrieved knowledge into training prompts with masked losses, RARE transforms learning objectives from rote memorization to contextualized reasoning, making implicit reasoning patterns salient. It enables models to bypass parameter-intensive memorization and prioritize the modeling of higher-order reasoning patterns. Extensive experiments demonstrate that lightweight RARE-trained models (e.g., Llama-3.1-8B and Qwen3-8B) could achieve state-of-the-art performance, even rivaling retrieval-augmented Gemini-3-Pro-Preview and GPT-5.1 with trillion parameters. RARE catalyzes a paradigm shift where maintainable external knowledge bases synergize with compact, reasoning-optimized models, collectively driving more scalable domain-specific intelligence.
Funding Information
  • Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM113)
  • National Natural Science Foundation of China (92470121)
  • National Natural Science Foundation of China (62402016)
  • National Key R&D Program of China (2024YFA1014003)
  • Zhongguancun Academy (C20250204)
  • Zhongguancun Academy (C20250602)
  • Beijing Major Science and Technology Project (Z251100008125043)
  • Beijing Major Science and Technology Project (Z251100008425023)
  • Shanghai Municipal Special Program for Basic Research on General AI Foundation Models (2025SHZDZX025G05)

This publication has 14 references indexed in Scilit: