Aziz Boxwala

BACKGROUND Conversational health AI is increasingly being developed for patient-facing applications, but evaluating stochastic, multi-turn systems before human studies remains challenging. Simulated patients could provide repeatable test environments, but their usefulness depends on whether the simulated patient itself reliably preserves the behavioral characteristics defining the test case. OBJECTIVE We evaluated an evidence-grounded simulated-patient framework for testing conversational health AI, using medication adherence as a behavioral test case. Specifically, we examined whether explicit behavioral representation and structured operationalization improve simulation fidelity beyond the same evidence supplied as prose or conventional patient prompting. METHODS Behavioral profiles for South Asian and Chinese simulated patients were derived from independent systematic reviews. Across 80 standardized multi-turn conversations (880 patient responses), we compared structured evidence-grounded simulation with the same evidence supplied as prose, ad hoc prompting, and web-informed patient construction. Automated measures assessed behavioral grounding, recovery of assigned behaviors, fidelity to assigned behavioral characteristics, unsupported barriers, conversation behavior, and correspondence with an independent corpus of published patient utterances. RESULTS Structured operationalization improved fidelity of generated behavior to the intended evidence-derived patient state. In the primary paired ablation, structured operationalization increased mean conversation-level behavioral groundedness compared with the same evidence supplied as prose (0.850 vs 0.732; mean paired difference 0.117, 95% CI 0.077-0.157; Cohen dz=1.376; P<.001). Structure did not improve every measure: prose produced better recovery of assigned COM-B dimensions (F1-score: 0.641 vs 0.503; P<.001), and other outcomes varied by evidence source. CONCLUSIONS Evidence-grounded simulated patients provide a computationally evaluable approach for creating repeatable behavioral test cases for conversational health AI. Explicit structure improved behavioral grounding beyond evidence content alone but did not uniformly improve all fidelity measures. This computational stage can support pre-human testing while identifying where subsequent expert and patient validation remains necessary.

Objectives: Advances in informatics research come from academic, nonprofit, and for-profit industry organizations, and from academic-industry partnerships. While scientific studies of commercial products may offer critical lessons for the field, manuscripts authored by industry scientists are sometimes categorically rejected. We review historical context, community perceptions, and guidelines on informatics authorship. Process: We convened an expert panel at the American Medical Informatics Association 2022 Annual Symposium to explore the role of industry in informatics research and authorship with community input. The panel summarized session themes and prepared recommendations. Conclusions: Authorship for informatics research, regardless of affiliation, should be determined by International Committee of Medical Journal Editors uniform requirements for authorship. All authors meeting criteria should be included, and categorical rejection based on author affiliation is unethical. Informatics research should be evaluated based on its scientific rigor; all sources of bias and conflicts of interest should be addressed through disclosure and, when possible, methodological mitigation.