Characterizing Deep Research: A Benchmark and Formal Definition
Authors:
- Abhinav JavaAJAbhinav Java
- Ashmit KhandelwalAKAshmit Khandelwal
- Sukruta MidigeshiSMSukruta Midigeshi
- Aaron HalfakerAHAaron Halfaker
- Amit DeshpandeADAmit Deshpande
- Navin GoyalNGNavin Goyal
- Ankur GuptaAGAnkur Gupta
- Nagarajan NatarajanNNNagarajan Natarajan
- Amit SharmaASAmit Sharma
Preprint
Abstract
Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of \textit{deep research} -- a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensive problems is poorly understood. In this paper, we propose a formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research is not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and reasoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search-separating the reasoning challenge from surface-level report generation. Based on this formulation, we propose a diverse, challenging benchmark LiveDRBench with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, F1 score ranges between 0.02 and 0.72 for any sub-category. OpenAI's model performs the best with an overall F1 score of 0.55. Analysis of reasoning traces reveals the distribution over the number of referenced sources, branching, and backtracking events executed by current DR systems, motivating future directions for improving their search mechanisms and grounding capabilities. The benchmark is available at https://github.com/microsoft/LiveDRBench.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Synthesizing scientific literature with retrieval-augmented language modelsNature, 2026
- DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments2025
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation2025
- Analysis of Plan-based Retrieval for Grounded Text Generation2024
- Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models2024