Evaluating AI-Generated Advice: Source Cues, Argument Quality, and Individual Differences in Credibility and Adherence
Authors:
- Elodie AndrieuEAElodie Andrieu
- Aurelie PistonoAPAurelie Pistono
- Franck AmadieuFAFranck Amadieu
Preprint
- 26 June 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
As generative AI systems are increasingly used to provide advice in complex situations, understanding how recipients evaluate such messages has become essential. The present study examined how perceived credibility and adherence depend on message source (AI vs. human), message quality (well-written vs. poorly written messages), and individual dispositions toward AI and intellectual humility.A sample of 175 participants evaluated ten scenarios describing challenging student situations, each followed by a response attributed either to a human expert or to ChatGPT and manipulated to display either high or low content quality. Data were analyzed using cumulative link mixed models (CLMM) to account for the ordinal nature of the dependent variables and the repeated-measures design.The results show no main effect of message source on credibility or adherence: responses attributed to AI are evaluated similarly to those attributed to a human expert. Message quality increases perceived credibility, with well-written messages judged as more credible, while no main effect of quality emerges for adherence. In contrast, attitudes toward AI modulate these evaluations: more favorable attitudes heighten sensitivity to message quality. Intellectual humility does not emerge as a significant main predictor, although exploratory analyses suggest a secondary role.Overall, these results indicate that the evaluation of advice relies primarily on message content and individual expectations rather than solely on source attribution. They suggest that fostering critical engagement with these systems is a key lever for their effective integration in advisory contexts.Keywords
This publication has 18 references indexed in Scilit:
- Can ChatGPT provide a better support: a comparative analysis of ChatGPT and dataset responses in mental health dialoguesCurrent Psychology, 2024
- Factors influencing online health information credibility: a meta-analysisAslib Journal of Information Management, 2024
- Is artificial intelligence more persuasive than humans? A meta-analysisJournal of Communication, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- The Roles of Personality Traits, AI Anxiety, and Demographic Factors in Attitudes toward Artificial IntelligenceInternational Journal of Human–Computer Interaction, 2022
- Is Automated Journalistic Writing Less Biased? An Experimental Test of Auto-Written and Human-Written News StoriesJournalism Practice, 2019
- Exploring early adolescents’ evaluation of academic and commercial online resources related to healthReading and Writing, 2017
- The Development and Validation of the Comprehensive Intellectual Humility ScaleJournal of Personality Assessment, 2015
- College students’ credibility judgments and heuristics concerning WikipediaInformation Processing & Management, 2013
- Heuristic processing can bias systematic processing: Effects of source credibility, argument ambiguity, and task importance on attitude judgment.Journal of Personality and Social Psychology, 1994