Mingshu Li

As large language models (LLMs) increasingly power financial conversational assistants, detecting investment advice in natural language has become a critical regulatory and safety challenge. Unlike explicit single-turn recommendations, investment advice often emerges implicitly and progressively over multi-turn conversations, shaped by user intent, delivery style, and contextual buildup. We introduce the first comprehensive framework for detecting both sequential and contextual investment advice in dialogue settings. Central to our approach is a modular, top-down synthetic data generation pipeline that systematically spans a taxonomy of advisory trajectories using an axis-bucket-descriptor schema. We simulate both benign and high-risk conversations across diverse user personas, financial instruments, and risk tiers, and evaluate detection capabilities using LLM-as-a-Judge with entropy-based uncertainty diagnostics. Our results show that contextual advisory detection remains substantially more challenging than sequential cases, even under few-shot prompting with tailored demonstrations. This work lays the foundation for scalable guardrail development in highstakes, compliance-sensitive domains, where advisory risk is often subtle, distributed, and perception-dependent.

Large language models (LLMs) reportedly 'reason' with increasing sophistication, but whether they 'know' when they are wrong remains poorly understood. We study epistemic calibration, the alignment between expressed confidence and empirical correctness, across approximately 30 contemporary models and configurable reasoning-effort budgets, using XFinBench (graduate-level financial reasoning) and SQuAD v2 (answerable and unanswerable questions). Reported confidence stays high relative to accuracy, with weak discrimination between correct and incorrect or unsupported answers. Miscalibration follows a Dunning-Kruger-like pattern: weaker models are often strongly overconfident, while frontier models are mildly underconfident. Reasoning effort significantly improves accuracy but has no detectable effect on the confidence-accuracy gap or ECE; stable between-model differences explain over 96% of calibration variance, versus a negligible effort effect. Calibration deteriorates most on scenario planning and numerical modeling. On SQuAD v2, models answer four in five unanswerable questions and remain near-maximally confident even when no answer is supported; higher reasoning effort yields no robust gain in calibration or abstention after multiplicity correction. We discuss implications for evaluation, deployment, and the design of calibration-aware, boundary-aware financial reasoning systems.

Retrieval-augmented generation (RAG) is increasingly used in financial applications where generated answers must be grounded in specific documents, filings, or disclosures. Yet even with relevant context, answers may contradict, overextend, or misinterpret the supplied evidence. This paper presents a comparative study of intrinsic hallucination detection for financial RAG, where the task is to determine whether an answer is faithful to a supplied question-context pair. We evaluate DSPy-based prompt optimization, fine-tuned detectors, encoder-based detectors, and LLM-as-Judge frameworks across public benchmarks and finance-specific settings. To examine finance-specific behavior, we construct FinDER-HD, a balanced hallucination-detection benchmark derived from FinDER by synthesizing unfaithful counterparts for faithful financial question-context-answer examples. We further evaluate filing-scale robustness on PHANTOM, which varies context length, evidence position, and filing type. Results show that detector performance varies substantially across datasets, financial topics, reasoning types, and context regimes. DSPy-based prompt optimization is competitive under limited supervision, but robustness differs markedly by optimization strategy: demonstration-heavy methods perform strongly in short contexts yet degrade more sharply as context grows, whereas COPRO and especially GEPA remain substantially more stable. Selected LLM-as-Judge methods also perform strongly in finance-specific settings, but their operating behavior depends on threshold choice and prediction bias. These findings motivate deployment-oriented evaluation of financial RAG guardrails beyond aggregate F1, including finance-specific robustness, filing-scale stress testing, prediction bias, threshold calibration, and contextwindow feasibility.

Due to the dynamic nature of financial markets, maintaining models that produce precise predictions over time is difficult. Often the goal isn’t just point prediction but determining uncertainty. Quantifying uncertainty, especially the aleatoric uncertainty due to the unpredictable nature of market drivers, helps investors understand varying risk levels. Recently, quantile regression forests (QRF) have emerged as a promising solution: Unlike most basic quantile regression methods that need separate models for each quantile, quantile regression forests estimate the entire conditional distribution of the target variable with a single model, while retaining all the salient features of a typical random forest. We introduce a novel approach to compute quantile regressions from random forests that leverages the proximity (i.e., distance metric) learned by the model and infers the conditional distribution of the target variable. We evaluate the proposed methodology using publicly available datasets and then apply it towards the problem of forecasting the average daily volume of corporate bonds. We show that using quantile regression using Random Forest proximities demonstrates superior performance in approximating conditional target distributions and prediction intervals to the original version of QRF. We also demonstrate that the proposed framework is significantly more computationally efficient than traditional approaches to quantile regressions.