Azra Ismail

BACKGROUND Artificial Intelligence (AI)-enabled systems are increasingly being positioned to address structural gaps in healthcare access, including in low- and middle-income countries (LMICs) facing workforce shortages and limited specialist availability. In particular, Large Language Models (LLMs), a type of AI model that generates and interprets text (e.g., GPT, Claude, and Gemini), are rapidly integrated into healthcare systems. Most research has focused on implementation in well-resourced settings, leaving LLM integration in low-and middle-income countries poorly understood. This study examines real-world LLM adoption in global health practice, focusing on the motivations, risks, and the infrastructural and organizational factors that shape implementation in low-and middle-income countries. OBJECTIVE This study aimed to examine how LLMs are implemented in real-world global health settings and to explore the motivations, perceived opportunities and risks, and organizational and infrastructural factors that shape their integration into health workflows and decision-making. METHODS We conducted semi-structured interviews with 17 professionals involved in the design, engineering, implementation, study, and oversight of LLM-based tools in global health. We recruited participants using purposive and snowball sampling to capture variation across role type and geographies, including South Asia, the Eastern Mediterranean, Africa, and multi-country global programs. We analyzed data using reflexive thematic analysis. RESULTS Five themes emerged. First, participants described LLMs as useful for reducing provider workload and expanding access to health information, but adoption was also shaped by funding pressures. Second, implementation was constrained by inadequate infrastructure, unreliable connectivity, reduced accuracy in low-resource languages, and data limitations. Third, local system integration required extensive sociotechnical effort, including co-design and linguistic customization. Fourth, organizations relied on multiple safety architectures and hybrid care pathways to safely deploy LLM tools in healthcare contexts. Fifth, organizations lacked clear outcome measures linking LLM use to health outcomes. CONCLUSIONS LLM implementation in global health is shaped by healthcare gaps, funding availability, and substantial socio-technical work, including workflow integration, human oversight, and investment in local language data. Gaps in accountability, safety evaluation, and outcome measurement mean limited evidence for safe and equitable scale-up, highlighting the need for stronger governance, safety, and outcome-focused evaluation frameworks. CLINICALTRIAL NA

Intimate health remains a taboo subject across many cultural contexts globally, limiting opportunities for open discussion and support. HCI research has established that religious and cultural identities and values shape discussions on intimate health, yet these dimensions are rarely embedded in the design of digital health platforms. In this paper, we adopt an Islamic Feminist lens to design a virtual environment to foster conversations about intimate health among Arab Muslim women. We conducted nineteen interviews, followed by four focus groups on an avatar-based platform (Gather.town) modeled on the majlis—a traditional Arab gathering space. While interviews revealed significant barriers to discussing intimate health, focus groups demonstrated how anonymous, avatar-based interactions in a familiar and comfortable setting can support open discussions. Our research thus advances HCI research on safe digital spaces for sensitive health conversations, and offers actionable strategies for embedding feminist, culturally, and religiously resonant design into digital health platforms.

Background: Sexual and reproductive health (SRH) remains a stigmatized and taboo topic globally, limiting access to reliable information. These challenges are heightened in the Global South, where linguistic and cultural diversity further complicates information access. In India (the study context), many individuals express SRH concerns in code-mixed language, such as Hinglish (code-mixed Hindi and English), and use colloquial terms. Large language models (LLMs) could help answer SRH questions, but most are trained for English and may perform poorly on code-mixed text and miss cultural nuances. Our research aims to address this gap by assessing the current state of LLMs in understanding user intent in SRH queries for a low-resource language. Objective: We evaluate the effectiveness of proprietary, multilingual open-weight, and Indic LLMs in zero-shot settings for identifying user intent in code-mixed Hinglish SRH queries. Our goal is to assess how well LLMs assign correct labels in a 2-level hierarchical classification (topic and subtopic). We take a hierarchical approach because SRH queries are complex and context-dependent; flat labels may obscure clinically important distinctions and lead to misdirected guidance. We also characterize common error types driving misclassification. Methods: We analyzed 4161 deidentified questions about SRH in Hinglish, collected by our partner nonprofit organization (Myna Mahila Foundation) in an underserved community in urban Mumbai. Queries were annotated into 8 topics and 40 subtopics using a hierarchical framework that captured linguistic, cultural, and contextual variation. We evaluated proprietary, multilingual open-weight, and Indic-specific LLMs in zero-shot settings. Performance was measured using hierarchical F ₁ (h F ₁ ), Exact Match, and topic- and subtopic-level accuracy. Results: Proprietary models achieved the strongest results, with GPT-5 performing best overall (h F ₁ = 0.784). Among open-weight systems, Sarvam-M emerged as the top-performing Indic model (h F ₁ =0.757), ranking just below the top-performing proprietary model and performing comparably with Claude-3.5-Sonnet (0.745; Anthropic) as well as large multilingual systems such as Llama-3.3-70B-Instruct (0.742; Meta) and Gemma-3-27B-IT (0.739; Google). Other Indic models performed considerably lower (eg, Llama-3-Gaja-Hindi-8B [0.596; CognitiveLab], Krutrim-2-Instruct [0.558; OLA Krutrim Team], and Airavata [0.404; AI4Bharat]). Smaller multilingual open-weight models, including Mixtral-8 × 7B-Instruct (0.593), Llama-3.1-8B-Instruct (0.630), Gemma-2-9B-IT (0.657), consistently outperformed them, showing that parameter size alone does not explain performance gaps. While models generally captured broad topical intent, they frequently failed at fine-grained intent recognition, especially with euphemisms, colloquial expressions, and locally or culturally situated questions. Conclusions: Hierarchical classification revealed persistent gaps in how LLMs handle code-mixed queries. Proprietary models performed best, but Sarvam-M shows that open-weight Indic systems can achieve performance near state-of-the-art models when supported by robust training data, cultural adaptation, and appropriate scale. These findings highlight the potential of localized, culturally aligned models to advance linguistically inclusive artificial intelligence tools and expand equitable access to SRH information in underserved populations globally.

BACKGROUND Access to knowledge about sexual and reproductive health (SRH) remains stigmatized and taboo in many parts of the globe. In the Global South, information delivery is further complicated by linguistic and cultural diversity. For instance, in India (our study context), urban Hindi-speaking users frequently type text in Hinglish (code-mixed Hindi and English written in the Latin script) and use colloquial language to describe SRH concerns. Large language models (LLMs) could help answer SRH questions, but most systems are trained for English and struggle with code-mixed text and understanding cultural context. Our research aims to address this gap by focusing on the current state of LLMs in understanding user intent in SRH queries, for a low-resource language. OBJECTIVE This study evaluates the effectiveness of proprietary, multilingual open-weight, and Indic LLMs in zero-shot settings for identifying user intent in code-mixed Hinglish SRH queries. Our aim is to measure how well LLMs assign the correct label in a two-level hierarchical classification (topic then subtopic). We take a hierarchical approach because SRH queries are complex and context-dependent; flat labels could obscure clinically important distinctions and can lead to misdirected guidance. We also characterize common error types that drive misclassification. METHODS We analyzed 4,161 de-identified questions about SRH in Hinglish (Hindi written in Latin script), collected by our partner nonprofit health organization (Myna Mahila Foundation) in an underserved community in urban Mumbai. Queries were annotated into 8 topics and 40 subtopics using a hierarchical framework that captured linguistic, cultural, and context variation. We compared the performance of proprietary, multilingual open-weight, and Indic-specific LLMs in zero-shot settings. Performance was measured using hierarchical F1(hF1), exact match, and topic- and subtopic-level accuracy. RESULTS Proprietary models achieved the strongest results, with GPT-5 performing best overall (hF1 = 0.784). Among open-weight systems, Sarvam-M emerged as the top-performing Indic model (hF1 = 0.757), ranking just below proprietary models and even surpassing Claude-3.5-Sonnet (0.745) as well as large multilingual systems such as LLaMA-3.3-70B-Instruct (0.742) and Gemma-3-27B-IT (0.739). Other Indic models performed considerably lower (e.g., LLaMA3-Gaja-Hindi-8B, 0.596; Krutrim-2-Instruct, 0.558; Airavata, 0.404). Smaller multilingual open-weights models––including Mixtral-8x7B-Instruct (0.593), LLaMA-3.1-8B-Instruct (0.630), Gemma-2-9B-IT (0.657)––consistently outperformed them, showing that parameter size alone does not explain performance gaps. While models generally captured broad topical intent, they frequently failed at fine-grained intent recognition, especially with euphemisms, colloquial expressions, and local and culturally situated questions. CONCLUSIONS Hierarchical classification revealed persistent gaps in how LLMs handle code-mixed queries. Proprietary models performed best, but Sarvam-M shows that open-weight Indic systems can achieve near–state-of-the-art performance when supported by robust training data and cultural adaptation. Strengthening such localized fine-tuned models is essential for developing culturally informed, linguistically inclusive AI tools that can expand equitable access to SRH information in underserved populations globally.

The integration of Large Language Models (LLMs) into healthcare settings has gained significant attention, particularly for question-answering tasks. Given the high-stakes nature of healthcare, it is essential to ensure that LLM-generated content is accurate and reliable to prevent adverse outcomes. However, the development of robust evaluation metrics and methodologies remains a matter of much debate. We examine the performance of publicly available LLM-based chatbots for menopause-related queries, using a mixed-methods approach to evaluate safety, consensus, objectivity, reproducibility, and explainability. Our findings highlight the promise and limitations of traditional evaluation metrics for sensitive health topics. We propose the need for customized and ethically grounded evaluation frameworks to assess LLMs to advance safe and effective use in healthcare.