Developing an Automatic System for Classifying Chatter About Health Services on Twitter: Case Study for Medicaid (Preprint)
Preprint
- 18 December 2020
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND The wide adoption of social media in daily life renders it a rich and effective resource for conducting near real-time assessments of consumers’ perceptions of health services. However, its use in these assessments can be challenging because of the vast amount of data and the diversity of content in social media chatter. OBJECTIVE This study aims to develop and evaluate an automatic system involving natural language processing and machine learning to automatically characterize user-posted Twitter data about health services using Medicaid, the single largest source of health coverage in the United States, as an example. METHODS We collected data from Twitter in two ways: via the public streaming application programming interface using Medicaid-related keywords (Corpus 1) and by using the website’s search option for tweets mentioning agency-specific handles (Corpus 2). We manually labeled a sample of tweets in 5 predetermined categories or other and artificially increased the number of training posts from specific low-frequency categories. Using the manually labeled data, we trained and evaluated several supervised learning algorithms, including support vector machine, random forest (RF), naïve Bayes, shallow neural network (NN), k-nearest neighbor, bidirectional long short-term memory, and bidirectional encoder representations from transformers (BERT). We then applied the best-performing classifier to the collected tweets for postclassification analyses to assess the utility of our methods. RESULTS We manually annotated 11,379 tweets (Corpus 1: 9179; Corpus 2: 2200) and used 7930 (69.7%) for training, 1449 (12.7%) for validation, and 2000 (17.6%) for testing. A classifier based on BERT obtained the highest accuracies (81.7%, Corpus 1; 80.7%, Corpus 2) and F1 scores on consumer feedback (0.58, Corpus 1; 0.90, Corpus 2), outperforming the second best classifiers in terms of accuracy (74.6%, RF on Corpus 1; 69.4%, RF on Corpus 2) and F1 score on consumer feedback (0.44, NN on Corpus 1; 0.82, RF on Corpus 2). Postclassification analyses revealed differing intercorpora distributions of tweet categories, with political (400778/628411, 63.78%) and consumer feedback (15073/27337, 55.14%) tweets being the most frequent for Corpus 1 and Corpus 2, respectively. CONCLUSIONS The broad and variable content of Medicaid-related tweets necessitates automatic categorization to identify topic-relevant posts. Our proposed system presents a feasible solution for automatic categorization and can be deployed and generalized for health service programs other than Medicaid. Annotated data and methods are available for future studies. CLINICALTRIALKeywords
All Related Versions
- Published version: Journal of Medical Internet Research, 23 (5), e26616.
This publication has 37 references indexed in Scilit:
- National and Local Influenza Surveillance through Twitter: An Analysis of the 2012-2013 Influenza EpidemicPLOS ONE, 2013
- Social Media Adoption in Local Health Departments NationwideAmerican Journal of Public Health, 2013
- Lexical normalization for social media textACM Transactions on Intelligent Systems and Technology, 2013
- Sentiment Analysis and Opinion MiningSynthesis Lectures on Human Language Technologies, 2012
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Reviews, Reputation, and Revenue: The Case of Yelp.ComSSRN Electronic Journal, 2011
- Multi-Label ClassificationInternational Journal of Data Warehousing and Mining, 2007
- Long Short-Term MemoryNeural Computation, 1997
- Grounded Theory and Organizational ResearchThe Journal of Applied Behavioral Science, 1986
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968