Human-AI Interaction: Evaluating LLM Reasoning on Digital Logic Circuit included Graph Problems, in terms of creativity in design and analysis
Preprint
Abstract
Large Language Models (LLMs) are increasingly used by undergraduate students as on-demand tutors, yet their reliability on circuit- and diagram-based digital logic problems remains unclear. We present a human- AI study evaluating three widely used LLMs (GPT, Gemini, and Claude) on 10 undergraduate-level digital logic questions spanning non-standard counters, JK-based state transitions, timing diagrams, frequency division, and finite-state machines. Twenty-four students performed pairwise model comparisons, providing per-question judgments on (i) preferred model, (ii) perceived correctness, (iii) consistency, (iv) verbosity, and (v) confidence, along with global ratings of overall model quality, satisfaction across multiple dimensions (e.g., accuracy and clarity), and perceived mental effort required to verify answers. To benchmark technical validity, we applied an independent judge-based evaluation against official solutions for all ten questions, using strict correctness criteria. Results reveal a consistent gap between perceived helpfulness and formal correctness: for the most sequentially demanding problems (Q1- Q7), none of the evaluated LLMs matched the official answers, despite producing confident, well-structured explanations that students often rated favorably. Error analysis indicates that models frequently default to canonical textbook templates (e.g., standard ripple counters) and struggle to translate circuit structure into exact state evolution and timing behavior. These findings suggest that, without verification scaffolds, LLMs may be unreliable for core digital logic topics and can inadvertently reinforce misconceptions in undergraduate instruction.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Trust Calibration for Joint Human/AI Decision-Making in Dynamic and Uncertain Contexts2025
- The Impact of Artificial General Intelligence-Assisted Project-Based Learning on Students’ Higher Order Thinking and Self-EfficacyIEEE Transactions on Learning Technologies, 2024
- Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT2023
- Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up SurveyIEEE Access, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI adviceComputers in Human Behavior, 2022
- A Survey on Visual Question Answering2021
- Empirical Evaluations of Framework for Adaptive Trust Calibration in Human-AI CooperationIEEE Access, 2020
- The Affordances and Constraints of Diagrams on Students' Reasoning about State Machines2017
- Flip-Flops in Students' Conceptions of StateIEEE Transactions on Education, 2011