From Verification Burden to Trusted Collaboration: Design Goals for LLM-Assisted Literature Reviews
Authors:
- Toby Jia-Jun LiTLToby Jia-Jun Li
Preprint
Abstract
Large Language Models (LLMs) are increasingly embedded in academic writing practices. Although numerous studies have explored how researchers employ these tools for scientific writing, their concrete implementation, limitations, and design challenges within the literature review process remain underexplored. In this paper, we report a user study with researchers across multiple disciplines to characterize current practices, benefits, and \textit{pain points} in using LLMs to investigate related work. We identified three recurring gaps: (i) lack of trust in outputs, (ii) persistent verification burden, and (iii) requiring multiple tools. This motivates our proposal of six design goals and a high-level framework that operationalizes them through improved related papers visualization, verification at every step, and human-feedback alignment with generation-guided explanations. Overall, by grounding our work in the practical, day-to-day needs of researchers, we designed a framework that addresses these limitations and models real-world LLM-assisted writing, advancing trust through verifiable actions and fostering practical collaboration between researchers and AI systems.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information Systems, 2025
- Use of large language models as artificial intelligence tools in academic research and publishing among global clinical researchersScientific Reports, 2024
- Large Language Models for Literature Reviews - An Exemplary Comparison of LLM-based Approaches with Manual Methods2024
- Artificial intelligence for literature reviews: opportunities and challengesArtificial Intelligence Review, 2024
- The Howard‐Harvard effect: Institutional reproduction of intersectional inequalitiesJournal of the Association for Information Science and Technology, 2024
- Can Generative AI improve social science?Proceedings of the National Academy of Sciences, 2024
- The application of large language models in medicine: A scoping reviewiScience, 2024
- Three ways ChatGPT helps me in my academic writingNature, 2024
- Generative Agents: Interactive Simulacra of Human Behavior2023
- Target-aware Abstractive Related Work Generation with Contrastive Learning | Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval2022