A dataset for medical instructional video classification and question answering
Open Access
- 22 March 2023
- journal article
- research article
- Published by Springer Nature in Scientific Data
- Vol. 10 (1) , 1-16
Abstract
This paper introduces a new challenge and datasets to foster research toward designing systems that can understand medical videos and provide visual answers to natural language questions. We believe medical videos may provide the best possible answers to many first aid, medical emergency, and medical education questions. Toward this, we created the and datasets and introduce the tasks of Medical Video Classification (MVC) and Medical Visual Answer Localization (MVAL), two tasks that focus on cross-modal (medical language and medical video) understanding. The proposed tasks and datasets have the potential to support the development of sophisticated downstream applications that can benefit the public and medical practitioners. Our datasets consist of 6,117 fine-grained annotated videos for the MVC task and 3,010 questions and answers timestamps from 899 videos for the MVAL task. These datasets have been verified and corrected by medical informatics experts. We have also benchmarked each task with the created and datasets and propose the multimodal learning methods that set competitive baselines for future research.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Gated Self-Matching Networks for Reading Comprehension and Question Answering2017
- MovieQA: Understanding Stories in Movies through Question-Answering2016
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Glove: Global Vectors for Word Representation2014
- Convolutional Neural Networks for Sentence Classification2014
- NLTK2004
- Long Short-Term MemoryNeural Computation, 1997
- Support-vector networksMachine Learning, 1995
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960