An LLM-Based Multi-Path Question Answering System with XGBoost Routing and Threshold-Based Refusal
Open Access
- 27 April 2026
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 15 (9) , 1845
Abstract
In conventional question answering systems, general-purpose large language models (LLMs), despite their strong capabilities in language understanding and generation, exhibit notable limitations in scenarios with stringent factuality requirements. Their outputs often lack explicit evidential grounding, making them prone to hallucinations and inconsistent responses. Moreover, LLMs do not inherently guarantee determinism when performing operations over structured data—such as aggregation, conditional filtering, and cross-field constraints—thereby undermining result consistency and reliability. To address these issues, we propose an internal-data-first framework for controllable question answering in high-risk scenarios. The framework categorizes knowledge sources into unstructured documents and structured data, enabling evidence-constrained generation via retrieval-augmented generation (RAG) and database-backed, verifiable query execution via restricted, read-only structured queries. In addition, an UNK branch is introduced as a safe degradation mechanism that triggers refusal when inputs lack sufficient evidence, exceed system capability boundaries, or fail to meet confidence requirements, thereby suppressing hallucinations and unauthorized generation. To enable controlled selection among the two execution pathways (RAG/SQL) and the safety degradation branch (UNK) at the system level, we design a learned router based on XGBoost with confidence-thresholded selective prediction, which preferentially activates UNK refusals for low-confidence or out-of-distribution inputs. We validate the proposed framework using a graduate admissions consultation system as an exemplar application, constructing both a document knowledge base and structured score tables, and conducting controlled comparisons across multiple system variants with multi-metric evaluations. Experimental results indicate that, under the current controlled evaluation setting, the proposed framework exhibits relatively stable behavior under complex query formulations and demonstrates practical engineering potential in high-risk vertical-domain question answering scenarios.Keywords
Funding Information
- National Natural Science Foundation of China (71762010)
- National Natural Science Foundation of China (Hnjg2024ZD-26)
- 2025 Hainan Provincial Postgraduate Innovation Research Project (Qhys2024-389)
This publication has 18 references indexed in Scilit:
- A Survey on Large Language Models for Code GenerationACM Transactions on Software Engineering and Methodology, 2026
- Instruction Tuning for Large Language Models: A SurveyACM Computing Surveys, 2026
- Retrieval-Augmented Generation for AI-Generated Content: A SurveyData Science and Engineering, 2026
- Advancing Financial Engineering with Foundation Models: Progress, Applications, and ChallengesEngineering, 2025
- Cracking SQL Barriers: An LLM-based Dialect Translation SystemProceedings of the ACM on Management of Data, 2025
- Retrieval augmented generation for large language models in healthcare: A systematic reviewPLOS Digital Health, 2025
- Retrieval-augmented generation for educational application: A systematic surveyComputers and Education: Artificial Intelligence, 2025
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information Systems, 2025
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challengesVicinagearth, 2024
- GPT-4 Technical ReportarXiv, 2023