Nikhil Jaiswal

Clinical AI is advancing faster than the infrastructure used to evaluate it. Current practice often relies on static benchmarks and aggregate metrics that incompletely capture how models behave across clinical settings, populations, and workflows. As a result, bias, shortcut learning, poor generalizability, and deployment-specific failure modes may remain undetected until clinical use. We introduce Distributed Open Justice Oversight (DOJO), an open-source, community-driven platform for adversarial evaluation of health AI systems. DOJO is designed to sit between data access and clinical deployment as a unified evaluation layer for continuous, system-level assessment across modalities and tasks. It integrates three evaluation planes: a data plane for surfacing dataset biases and shortcuts; a model plane that probes robustness and failure modes; and a clinical workflow plane, which simulates clinical integration to gauge deployment readiness, including computational performance and pipeline interoperability. DOJO looks past model-centric metrics. It evaluates the full technical system and its intended clinical use context by jointly analyzing the clinical workflow, AI model, data pipeline, and human interaction layer. Agent-based evaluation runs alongside human-in-the-loop validation so that stress testing and feedback can iterate quickly. We hypothesize that community-centered, agent-assisted evaluation can detect clinically meaningful failures that conventional benchmarking may miss. The goal is to make evaluation continuous and participatory so that translation of AI into clinical practice can be safer and more equitable.

Background: Acute infectious symptoms are a leading cause of pediatric emergency department visits in Canada, many of which are low acuity and could be safely managed at home. Artificial intelligence (AI) chatbots offer a promising avenue for delivering accessible, evidence-based guidance to support families in managing these symptoms. Objective: This study aims to adapt and coconstruct CHAMP (CHatbot to Assist the Management of Pediatric patients), an AI chatbot to support patients and families with acute pediatric infectious symptoms. CHAMP aims to deliver timely, tailored, and validated health information to support safe at-home self-management and informed care-seeking. Methods: This multiphase, mixed methods participatory study will be conducted at the Montreal Children’s Hospital in Montreal, Quebec, Canada. A coconstruction committee comprised of youth, parents, caregivers, and partners will be engaged as coresearchers. Eligible participants will include (1) youth aged 14‐17 years and (2) parents and caregivers of children aged 0‐17 years. The study comprises 5 phases. Phase 1 involves a qualitative needs assessment using focus groups with 20 participants to explore their informational needs, preferences, and concerns regarding pediatric infections and the use of AI chatbots. Phase 2 focuses on coconstructing and validating CHAMP’s knowledge database through 3‐5 workshops. Coresearchers will review pediatric clinical guidelines, map care questions and decision-making processes, and shape CHAMP’s conversational framework. Phase 3 consists of iterative prototyping and testing through 3‐5 workshops. Coresearchers will engage in prototyping and scenario testing, alongside preliminary usability and acceptability assessments. Phase 4 examines equity and accessibility through focus groups with 20 participants at risk of digital exclusion, as well as multilingual evaluation of an automated large language model–based translation layer. Phase 5 uses collaborative ethnography to explore the process of participatory coconstruction and its impact on CHAMP’s design. Results: Funding was secured in 2024, and Research Ethics Board approval was obtained in December 2024. As of December 2025, the coconstruction committee is being assembled, and Phase 1 recruitment is underway. Conclusions: This study will produce a functioning CHAMP prototype grounded in participatory, equitable, and responsible pediatric AI development. Findings will inform usability testing and an implementation-effectiveness evaluation, contributing to best practices for pediatric-centered AI health tools. By providing timely, tailored, and validated health information on acute infections, CHAMP may support safe at-home self-management, reduce preventable emergency department visits, ensure at-risk children are directed to appropriate care, and improve patient and family health care experiences.

BACKGROUND Acute infectious symptoms are leading causes of pediatric emergency department (ED) visits in Canada, many of which are low-acuity and could be safely managed at home. Artificial intelligence (AI) chatbots offer a promising avenue for delivering accessible, evidence-based guidance to support families in managing these symptoms. OBJECTIVE To adapt and co-construct CHAMP (CHatbot to Assist the Management of Pediatric patients), an AI chatbot to support patients and families with acute pediatric infectious symptoms. CHAMP aims to deliver timely, tailored, and validated health information to support safe at-home self-management and informed care-seeking. METHODS This multi-phase, mixed-methods participatory study will be conducted at the Montreal Children’s Hospital in Montreal, Canada. A Co-Construction Committee comprised of youth, parents, caregivers, and partners will be engaged as co-researchers. Eligible participants will include: (1) youth aged 14-17 years and (2) parents and caregivers of children aged 0-17 years. The study comprises five phases. Phase 1 involves a qualitative needs assessment using focus groups with 20 participants to explore informational needs, preferences, and concerns about pediatric infections and AI chatbot use. Phase 2 focuses on co-constructing and validating CHAMP’s knowledge database through 3-5 workshops. Co-researchers will review pediatric clinical guidelines, map care questions and decision-making processes, and shape CHAMP’s conversational framework. Phase 3 consists of iterative prototyping and testing through 3-5 workshops. Co-researchers will engage in prototyping and scenario testing, alongside preliminary usability and acceptability assessments. Phase 4 examines equity and accessibility through focus groups with 20 participants at risk of digital exclusion as well as multilingual evaluation of an automated large language model-based translation layer. Phase 5 employs collaborative ethnography to explore the process of participatory co-construction and its impact on CHAMP’s design. RESULTS Funding was secured in 2024 and REB approval was obtained in December 2024. As of December 2025, the Co-Construction Committee is being assembled and Phase 1 recruitment is underway. CONCLUSIONS This study will produce a functioning CHAMP prototype grounded in participatory, equitable, and responsible pediatric AI development. Findings will inform usability testing and an implementation-effectiveness evaluation, contributing to best practices for co-constructed, pediatric-centered AI health tools. By providing timely, tailored, and validated health information on acute infections, CHAMP may support safe at-home self-management, reduce preventable ED visits, ensure at-risk children are directed to appropriate care, and improve patient and family healthcare experiences. CLINICALTRIAL NCT05789901

Uses of large language models (LLMs) in health chatbots are expanding into high-stakes clinical contexts, heightening the need for tools that are evidence-based, accountable, accurate, and patient-centred. This conceptual, practice-informed Perspective reflects on engaging patients and non-academic partners for the responsible integration of LLMs, grounded in the co-construction of MARVIN (for people living with HIV) and in an emerging collaboration with MIT Critical Data. Organised by the Software Development Life Cycle, we describe: conception/needs assessment with patient partners to identify use cases, acceptable trade-offs, and privacy expectations; development that prioritises grounding via vetted sources, structured human feedback, and data-validation committees including patient partners; testing and evaluation using patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) chosen in collaboration with patients to capture usability, acceptability, trust, and perceived safety, alongside task performance and harmful-output monitoring; and implementation via diverse governance boards, knowledge-mobilisation materials to set expectations, and risk-management pathways for potentially unsafe outputs. Based on our experience with MARVIN, we recommend early and continuous engagement of patients and non-academic partners, fair compensation, shared decision-making power, transparent decision logging, and inclusive, adaptable governance that can evolve with changing models and standards. These lessons highlight how patient partnership can directly shape chatbot design and oversight, helping teams align LLM-enabled tools with patient-centred goals while building accountable, safe, and equitable systems.