Laypeople’s Use of and Attitudes Toward Large Language Models and Search Engines for Health Queries: Survey Study (Preprint)
Preprint
- 14 July 2024
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Laypeople have easy access to health information through large language models (LLMs), such as ChatGPT, and search engines, such as Google. Search engines transformed health information access, and LLMs offer a new avenue for answering laypeople’s questions. OBJECTIVE We aimed to compare the frequency of use and attitudes toward LLMs and search engines as well as their comparative relevance, usefulness, ease of use, and trustworthiness in responding to health queries. METHODS We conducted a screening survey to compare the demographics of LLM users and nonusers seeking health information, analyzing results with logistic regression. LLM users from the screening survey were invited to a follow-up survey to report the types of health information they sought. We compared the frequency of use of LLMs and search engines using ANOVA and Tukey post hoc tests. Lastly, paired-sample Wilcoxon tests compared LLMs and search engines on perceived usefulness, ease of use, trustworthiness, feelings, bias, and anthropomorphism. RESULTS In total, 2002 US participants recruited on Prolific participated in the screening survey about the use of LLMs and search engines. Of them, 52% (n=1045) of the participants were female, with a mean age of 39 (SD 13) years. Participants were 9.7% (n=194) Asian, 12.1% (n=242) Black, 73.3% (n=1467) White, 1.1% (n=22) Hispanic, and 3.8% (n=77) were of other races and ethnicities. Further, 1913 (95.6%) used search engines to look up health queries versus 642 (32.6%) for LLMs. Men had higher odds (odds ratio [OR] 1.63, 95% CI 1.34-1.99; P<.001) of using LLMs for health questions than women. Black (OR 1.90, 95% CI 1.42-2.54; P<.001) and Asian (OR 1.66, 95% CI 1.19-2.30; P<.01) individuals had higher odds than White individuals. Those with excellent perceived health (OR 1.46, 95% CI 1.1-1.93; P=.01) were more likely to use LLMs than those with good health. Higher technical proficiency increased the likelihood of LLM use (OR 1.26, 95% CI 1.14-1.39; P<.001). In a follow-up survey of 281 LLM users for health, most participants used search engines first (n=174, 62%) to answer health questions, but the second most common first source consulted was LLMs (n=39, 14%). LLMs were perceived as less useful (P<.01) and less relevant (P=.07), but elicited fewer negative feelings (P<.001), appeared more human (LLM: n=160, vs search: n=32), and were seen as less biased (P<.001). Trust (P=.56) and ease of use (P=.27) showed no differences. CONCLUSIONS Search engines are the primary source of health information; yet, positive perceptions of LLMs suggest growing use. Future work could explore whether LLM trust and usefulness are enhanced by supplementing answers with external references and limiting persuasive language to curb overreliance. Collaboration with health organizations can help improve the quality of LLMs’ health output.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 27, e64290.
This publication has 17 references indexed in Scilit:
- Leveraging Large Language Models for Decision Support in Personalized OncologyJAMA Network Open, 2023
- Accuracy and Reliability of Chatbot Responses to Physician QuestionsJAMA Network Open, 2023
- Assessing ChatGPT’s capacity for clinical decision support in pediatrics: A comparative study with pediatricians using KIDMAP of Rasch analysisMedicine, 2023
- Large language models for oncological applicationsJournal of Cancer Research and Clinical Oncology, 2023
- Using AI-generated suggestions from ChatGPT to optimize clinical decision supportJournal of the American Medical Informatics Association, 2023
- Anthropomorphism brings us closer: The mediating role of psychological distance in User–AI assistant interactionsComputers in Human Behavior, 2021
- Design, development and evaluation of a human-computer trust scaleBehaviour & Information Technology, 2019
- Patients using the Internet to obtain health information: How this affects the patient–health professional relationshipPatient Education and Counseling, 2006
- Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide WebJAMA, 2002
- Development and validation of brief measures of positive and negative affect: The PANAS scales.Journal of Personality and Social Psychology, 1988