RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media
Preprint
Abstract
We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. Collected annotations (and scripts to assemble the dataset), are available at https://github.com/sominw/redhot.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Effects of misinformation on COVID-19 individual responses and recommendations for resilience of disastrous consequences of misinformationProgress in Disaster Science, 2020
- Trialstreamer: A living, automatically updated database of clinical trial reportsJournal of the American Medical Informatics Association, 2020
- Public Health and Online Misinformation: Challenges and RecommendationsAnnual Review of Public Health, 2020
- COMETA: A Corpus for Medical Entity Linking in the Social Media2020
- Evidence Inference 2.0: More Data, Better Models2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Using Online Health Communication to Manage Chronic Sorrow: Mothers of Children with Rare Diseases SpeakJournal of Pediatric Nursing, 2015
- A BRIEF HISTORY OF THE RANDOMIZED CONTROLLED TRIALHematology/Oncology Clinics of North America, 2000