Detecting Recovery Problems Just in Time: Application of Automated Linguistic Analysis and Supervised Machine Learning to an Online Substance Abuse Forum (Preprint)
Preprint
- 15 February 2018
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Online discussion forums allow those in addiction recovery to seek help through text-based messages, including when facing triggers to drink or use drugs. Trained staff (or “moderators”) may participate within these forums to offer guidance and support when participants are struggling but must expend considerable effort to continually review new content. Demands on moderators limit the scalability of evidence-based digital health interventions. OBJECTIVE Automated identification of recovery problems could allow moderators to engage in more timely and efficient ways with participants who are struggling. This paper aimed to investigate whether computational linguistics and supervised machine learning can be applied to successfully flag, in real time, those discussion forum messages that moderators find most concerning. METHODS Training data came from a trial of a mobile phone-based health intervention for individuals in recovery from alcohol use disorder, with human coders labeling discussion forum messages according to whether or not authors mentioned problems in their recovery process. Linguistic features of these messages were extracted via several computational techniques: (1) a Bag-of-Words approach, (2) the dictionary-based Linguistic Inquiry and Word Count program, and (3) a hybrid approach combining the most important features from both Bag-of-Words and Linguistic Inquiry and Word Count. These features were applied within binary classifiers leveraging several methods of supervised machine learning: support vector machines, decision trees, and boosted decision trees. Classifiers were evaluated in data from a later deployment of the recovery support intervention. RESULTS To distinguish recovery problem disclosures, the Bag-of-Words approach relied on domain-specific language, including words explicitly linked to substance use and mental health (“drink,” “relapse,” “depression,” and so on), whereas the Linguistic Inquiry and Word Count approach relied on language characteristics such as tone, affect, insight, and presence of quantifiers and time references, as well as pronouns. A boosted decision tree classifier, utilizing features from both Bag-of-Words and Linguistic Inquiry and Word Count performed best in identifying problems disclosed within the discussion forum, achieving 88% sensitivity and 82% specificity in a separate cohort of patients in recovery. CONCLUSIONS Differences in language use can distinguish messages disclosing recovery problems from other message types. Incorporating machine learning models based on language use allows real-time flagging of concerning content such that trained staff may engage more efficiently and focus their attention on time-sensitive issues.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 20 (6), e10136.
This publication has 58 references indexed in Scilit:
- Development of Telehealth Dialogues for Monitoring Suicidal Patients with Schizophrenia: Consumer FeedbackCommunity Mental Health Journal, 2013
- Managing the Personal Side of Health: How Patient Expertise Differs from the Expertise of CliniciansJournal of Medical Internet Research, 2011
- Random forests for verbal autopsy analysis: multisite validation study using clinical diagnostic gold standardsPopulation Health Metrics, 2011
- Computer-based interventions for drug use disorders: A systematic reviewJournal of Substance Abuse Treatment, 2011
- Supportive Accountability: A Model for Providing Human Support to Enhance Adherence to eHealth InterventionsJournal of Medical Internet Research, 2011
- Variable selection using random forestsPattern Recognition Letters, 2010
- Internet Treatment for Depression: A Randomized Controlled Trial Comparing Clinician vs. Technician AssistancePLOS ONE, 2010
- An online support group for problem drinkers: AlcoholHelpCenter.netPatient Education and Counseling, 2008
- Developing adaptive treatment strategies in substance abuse researchDrug and Alcohol Dependence, 2007
- Special issue on dialog systems for health communicationJournal of Biomedical Informatics, 2006