Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues
Top Cited Papers
Open Access
- 14 March 2023
- journal article
- research article
- Published by MDPI AG in Future Internet
- Vol. 15 (3) , 110
Abstract
Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)—an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics.Keywords
Funding Information
- European Commission (812882, F75F21001220007)
- ASTRID project (812882, F75F21001220007)
This publication has 29 references indexed in Scilit:
- The Motivational Interviewing Treatment Integrity Code (MITI 4): Rationale, Preliminary Reliability and ValidityJournal of Substance Abuse Treatment, 2016
- A Comparison of Natural Language Processing Methods for Automated Coding of Motivational InterviewingJournal of Substance Abuse Treatment, 2016
- Building a Motivational Interviewing Dataset2016
- ESWC’14 Challenge on Concept-Level Sentiment Analysis2014
- Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classificationImplementation Science, 2014
- Behavioral observation.2012
- Motivational Interviewing in Health Care: Helping Patients Change BehaviorCOPD: Journal of Chronic Obstructive Pulmonary Disease, 2008
- Long Short-Term MemoryNeural Computation, 1997
- The Measurement of Observer Agreement for Categorical DataBiometrics, 1977
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971