Japanese-Language AI Agent System for Human Papillomavirus Vaccine Infoveillance and Public Communication: Development and Feasibility Evaluation
Open Access
- 21 May 2026
- journal article
- Published by JMIR Publications Inc. in JMIR Infodemiology
- Vol. 6 (1) , e90295
Abstract
Background: Human papillomavirus (HPV) vaccine hesitancy remains a significant public health challenge in Japan, where proactive vaccination recommendations were suspended between 2013 and 2021. The resulting information gap between medical institutions and vaccine-hesitant populations is exacerbated by misinformation on social media platforms. Traditional public health communication strategies cannot address individual queries while simultaneously monitoring population-level discourse. Objective: This study aimed to develop and conduct a feasibility evaluation of a dual-purpose artificial intelligence agent system that delivers verified HPV vaccine information to the public through a conversational interface while generating infoveillance reports for medical institutions based on user interactions and social media discourse. Methods: We implemented a system with 3 components: a vector database integrating 139,803 documents, including academic papers, Japanese government sources, news media, and social media posts; a retrieval-augmented generation chatbot using a ReAct agent architecture with iterative multitool orchestration across 5 specialized knowledge sources; and an automated report generation system with modules for news analysis, research synthesis, social media sentiment analysis, including stance classification and topic modeling, and user interaction pattern identification. System performance was assessed using both automated and manual evaluation protocols on a scale from 0 to 5. Results: The entire system functioned as expected. For single-turn evaluation, the chatbot achieved mean scores of 4.83 (SD 0.67; 95% CI 4.71-4.93) for relevance, 4.89 (SD 0.53; 95% CI 4.79-4.97) for routing, 4.50 (SD 1.29; 95% CI 4.27-4.70) for reference quality, 4.90 (SD 0.62; 95% CI 4.78-4.99) for correctness, and 4.88 (SD 0.54; 95% CI 4.78-4.96) for professional identity, with an overall mean of 4.80. Multiturn evaluation yielded higher mean scores: 4.94 for context memory (SD 0.25; 95% CI 4.84-5.00) and an overall mean of 4.98, with topic centering and identity achieving 5.00. The report generation system achieved high scores across all sections: 4.83 for completeness (SD 0.37; 95% CI 4.73-4.94), 4.88 for correctness (SD 0.33; 95% CI 4.77-4.96), and 4.12 for helpfulness (SD 0.48; 95% CI 3.98-4.27). Reference validity achieved perfect scores (5.00) across all periods, with citation correctness averaging 4.21 (SD 0.58; 95% CI 3.96-4.46). Conclusions: This feasibility study demonstrated that an integrated artificial intelligence agent system can support both public HPV vaccine communication and social media infoveillance in a Japanese-language context. Prospective deployment with real users is needed to assess actual public health impact.Keywords
This publication has 17 references indexed in Scilit:
- Evaluation of Retrieval-Augmented Generation: A Survey2025
- Global burden of cervical cancer: current estimates, temporal trend and future projections based on the GLOBOCAN 2022Journal of the National Cancer Center, 2025
- Large language models in medicineNature Medicine, 2023
- Resumption of active recommendation of the human papillomavirus vaccine in Japan and future challenges for the National Immunization ProgramHuman Vaccines & Immunotherapeutics, 2022
- The online competition between pro- and anti-vaccination viewsNature, 2020
- Impact of HPV vaccine hesitancy on cervical cancer in Japan: a modelling studyThe Lancet Public Health, 2020
- Conversational agents in healthcare: a systematic reviewJournal of the American Medical Informatics Association, 2018
- Vaccine hesitancy: Definition, scope and determinantsVaccine, 2015
- Population-level impact and herd effects following human papillomavirus vaccination programmes: a systematic review and meta-analysisThe Lancet Infectious Diseases, 2015
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988