The quality of data-driven hypotheses generated by inexperienced clinical researchers: A case study
Preprint
Abstract
Objectives We invited inexperienced clinical researchers to analyze coded health datasets and develop hypotheses. We recorded and analyzed their hypothesis generation process. All the hypotheses generated in the process were rated by the same group of seven experts by using the same metrics. This case study examines the higher quality (i.e., higher ratings) and lower quality of hypotheses and participants who generated them. We characterized the contextual factors associated with the quality of hypotheses. Methods All participants (i.e., clinical researchers) completed a 2-hour study session to analyze data and generate scientific hypotheses using the think-aloud method. Participants’ screen activity and audio were recorded and transcribed. These transcriptions were used to measure the time used to generate each hypothesis and to code cognitive events (i.e., cognitive activities used when generating hypotheses, for example, “Seeking for Connection” describes an attempt to draw connections between data points). The hypothesis ratings by the expert panel were used as the quality of the hypotheses during the analysis. We analyzed the factors associated with (1) the five highest and (2) five lowest rated hypotheses and (3) the participants who generated them, including the number of hypotheses per participant, the validity of those hypotheses, the number of cognitive events used for each hypothesis, as well as the participant’s research experience and basic demographics. Results Participants who generated the five highest-rated hypotheses used similar lengths of time (difference 3:03), whereas those who generated the five lowest-rated hypotheses used more varying lengths of time (difference 7:13). Participants who generated the five highest-rated hypotheses also utilized slightly fewer cognitive events on average compared to the five lowest-rated hypotheses (4 per hypothesis vs. 4.8 per hypothesis). When we examine the participants (who generated the five highest and five lowest hypotheses) and their total hypotheses generated during the 2-hour study sessions, the participants with the five highest-rated hypotheses again had a shorter range of time per hypothesis on average (0:03:34 vs. 0:07:17). They (with the five highest ratings) used fewer cognitive events per hypothesis (3.498 vs. 4.626). They (with the five highest ratings) also had a higher percentage of valid rate (75.51% vs. 63.63%) and generally had more experience with clinical research. Conclusion The quality of the hypotheses was shown to be associated with the time taken to generate them, where too long or too short time to generate hypotheses appears to be negatively associated with the hypotheses’ quality ratings. Also, having more experience seems to positively correlate with higher ratings of hypotheses and higher valid rates. Validity is a quality dimension used by the expert panel during rating. However, we acknowledge that our results are anecdotal. The effect may not be simply linear, and future research is necessary. These results underscore the multi-factor nature of hypothesis generation.Keywords
This publication has 17 references indexed in Scilit:
- Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other toolsJournal of Clinical and Translational Science, 2024
- How do clinical researchers generate data-driven scientific hypotheses? Cognitive events using think-aloud protocolmedRxiv, 2023
- Remembrance of inferences past: Amortization in human hypothesis generationCognition, 2018
- MOLIERE2017
- Automated hypothesis generation based on mining scientific literature2014
- Cognitive Informatics in Health and Biomedicine2014
- The effects of subjective time pressure and individual differences on hypotheses generation and action prioritization in police investigations.Journal of Experimental Psychology: Applied, 2013
- Hypothesis Generation and the Coordination of Theory and Evidence in Novice Diagnostic ReasoningMedical Decision Making, 1993
- Domain Knowledge and Hypothesis Genenation in Diagnostic ReasoningMedical Decision Making, 1990
- Knowledge based solution strategies in medical reasoningCognitive Science, 1986