Generative AI for Intelligence Augmentation: Categorization and Evaluation Frameworks for Large Language Model Adaptation
Authors:
- Fairfield UniversityFUFairfield University
- Jie TaoJTJie Tao
- Lina ZhouLZLina Zhou
- The University of North CaorlinaTCThe University of North Caorlina
- Xing FangXFXing Fang
- Illinois State UniversityIUIllinois State University
Open Access
- 30 September 2024
- journal article
- Published by Association for Information Systems in AIS Transactions on Human-Computer Interaction
- Vol. 16 (3) , 364-387
Abstract
Generative AI (GenAI) has transformed how businesses operate and innovate and how individuals learn, live, and work. Large language models (LLMs), a specific type of GenAI, focus on generating human-like text based on user instructions. Like other types of GenAI, LLMs have received wide recognition for their potential to augment human intelligence, but several challenges hinder efforts to realize their full potential in practice. Some notable challenges include not adequately exploring LLM applications beyond chatbots and/or text generation, the difficulty in categorizing various LLM adaptation strategies (particularly regarding human interactions), and the lack of a reference framework for evaluating and selecting LLM adaptation strategies from a human-centered perspective. To address these challenges, we propose a categorization framework for LLM adaptation that features two human-centered dimensions and stage LLM adaptation with respect to when it interacts with human intelligence. Additionally, we introduce an evaluation framework that incorporates a human-centered perspective that goes beyond the common machine-centered measures. Our empirical investigations, in which we use text classification as use cases, not only demonstrate the application of these frameworks but also compare various adaptation strategies. These artifacts and findings provide fresh insights and practical recommendations for selecting effective adaptation strategies to improve the efficacy of LLMs for intelligence augmentation. We further identify future research issues to address current limitations and suggest improvements for the proposed frameworks.Keywords
This publication has 25 references indexed in Scilit:
- Enhancing Autonomous System Security and Resilience With Generative AI: A Comprehensive SurveyIEEE Access, 2024
- A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIMEAdvanced Intelligent Systems, 2024
- Sentiment Analysis in the Age of Generative AICustomer Needs and Solutions, 2024
- A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and ChallengesIEEE Access, 2024
- Generative AI and ChatGPT: Applications, challenges, and AI-human collaborationJournal of Information Technology Case and Application Research, 2023
- ChatGPT: Jack of all trades, master of noneInformation Fusion, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Modeling Human Driving Behavior Through Generative Adversarial Imitation LearningIEEE Transactions on Intelligent Transportation Systems, 2022
- Fleiss’ kappa statistic without paradoxesQuality & Quantity, 2014
- Good debt or bad debt: Detecting semantic orientations in economic textsJournal of the Association for Information Science and Technology, 2013