Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study
Top Cited Papers
Open Access
- 30 January 2024
- journal article
- research article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e51069
Abstract
Background: Sentiment analysis is a significant yet difficult task in natural language processing. The linguistic peculiarities of Cantonese, including its high similarity with Standard Chinese, its grammatical and lexical uniqueness, and its colloquialism and multilingualism, make it different from other languages and pose additional challenges to sentiment analysis. Recent advances in models such as ChatGPT offer potential viable solutions. Objective: This study investigated the efficacy of GPT-3.5 and GPT-4 in Cantonese sentiment analysis in the context of web-based counseling and compared their performance with other mainstream methods, including lexicon-based methods and machine learning approaches. Methods: We analyzed transcripts from a web-based, text-based counseling service in Hong Kong, including a total of 131 individual counseling sessions and 6169 messages between counselors and help-seekers. First, a codebook was developed for human annotation. A simple prompt (“Is the sentiment of this Cantonese text positive, neutral, or negative? Respond with the sentiment label only.”) was then given to GPT-3.5 and GPT-4 to label each message’s sentiment. GPT-3.5 and GPT-4’s performance was compared with a lexicon-based method and 3 state-of-the-art models, including linear regression, support vector machines, and long short-term memory neural networks. Results: Our findings revealed ChatGPT’s remarkable accuracy in sentiment classification, with GPT-3.5 and GPT-4, respectively, achieving 92.1% (5682/6169) and 95.3% (5880/6169) accuracy in identifying positive, neutral, and negative sentiment, thereby outperforming the traditional lexicon-based method, which had an accuracy of 37.2% (2295/6169), and the 3 machine learning models, which had accuracies ranging from 66% (4072/6169) to 70.9% (4374/6169). Conclusions: Among many text analysis techniques, ChatGPT demonstrates superior accuracy and emerges as a promising tool for Cantonese sentiment analysis. This study also highlights ChatGPT’s applicability in real-world scenarios, such as monitoring the quality of text-based counseling services and detecting message-level sentiments in vivo. The insights derived from this study pave the way for further exploration into the capabilities of ChatGPT in the context of underresourced languages and specialized domains like psychotherapy and natural language processing.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- GPT is an effective tool for multilingual psychological text analysisPsyArXiv, 2023
- A machine learning approach to identifying suicide risk among text-based crisis counseling encountersFrontiers in Psychiatry, 2023
- Detecting and Analyzing Suicidal Ideation on Social Media Using Deep Learning and Machine Learning ModelsInternational Journal of Environmental Research and Public Health, 2022
- The Opportunities and Challenges of the First Three Years of Open Up, an Online Text-Based Counselling Service for Youth and Young AdultsInternational Journal of Environmental Research and Public Health, 2021
- Experiences of Online Counseling Among Undergraduates in Some Ghanaian UniversitiesSage Open, 2020
- The State and Fate of Linguistic Diversity and Inclusion in the NLP World2020
- Undergraduate Student Experiences with Text-Based Online CounsellingThe British Journal of Social Work, 2017
- Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental HealthTransactions of the Association for Computational Linguistics, 2016
- Investigating individual online synchronous chat counselling processes and treatment outcomes for young peopleAdvances in Mental Health, 2014
- Sentiment Analysis for Cantonese Opinion Mining2013