Betina Idnay
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Betina Idnay has a total of 81 co-authors. Here are the top co-authors by number of shared publications:
- Chunhua Weng
Department of Biomedical Informatics, Columbia University, New York, USA
11 shared publications Karen Marder
Department of Neurology, Columbia University Irving Medical Center , New York, NY 10032
8 shared publications- Rebecca Schnall
Columbia University School of Nursing
5 shared publications - Yilu Fang
Department of Biomedical Informatics, Columbia University , New York, NY 10032
5 shared publications - Hao Liu
Montclair State University
3 shared publications Brenda Ruotolo
Institutional Review Board for Human Subjects Research, Columbia University, New York, NY, USA
2 shared publications
Latest Publications
- Open Access
Returning aggregate research results to participants: a scoping review of current practices, preferences and challenges
Objectives: To synthesise the current evidence about practices, preferences and challenges related to returning aggregate research results to participants, with implications for public health, health equity and policy development. Design: Scoping review conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews framework. Data sources: Four electronic databases—PubMed (National Library of Medicine), Excerpta Medica Database (Elsevier), Cumulative Index to Nursing and Allied Health Literature (Elton B. Stephens CO(mpany)) and Cochrane Library (Wiley)—were from inception to February 2025. Eligibility criteria: English-language, peer-reviewed articles reporting practices or preferences for returning aggregate clinical research results to participants. Studies reporting only individual/incidental findings, opinion pieces and non-original research were excluded. Data extraction and synthesis: Two reviewers independently screened studies and extracted data. Study quality was assessed using a modified Oxford Centre for Evidence-Based Medicine scale. Thematic synthesis identified patterns in dissemination methods, participant preferences and implementation barriers. Results: Of 272 articles screened, 12 published between 2002 and 2019 were included. They employed cross-sectional, qualitative and mixed-methods design across North America, Europe, Africa and Australia. Half focused on cancer; others addressed malaria, autism, hypothyroidism, HIV prevention and preterm labour. Most (8/12) included patients; others also included researchers, caregivers or mixed stakeholders. Participants were predominantly middle-aged or older, female and well-educated. Only two studies assessed literacy, both using unvalidated self-reports. Eight studies explored preferences without returning results; four implemented disseminations via mailed reports, printed summaries, in-person discussions or digital platforms. Participants favoured mailed letters and face-to-face meetings over online methods. Six themes emerged: receiving results as respect; preference for personalised, timely formats; importance of cultural tailoring; emotional and ethical considerations; institutional barriers; and community engagement as a facilitator. Conclusions: Returning aggregate research results to participants is ethically supported and strongly desired yet rarely practised. Bridging this gap requires clearer policy guidance, institutional support and equity-focused dissemination strategies. Health systems and researchers should adopt scalable, participant-centred approaches to fulfil ethical obligations and strengthen public trust in science.
Open AccessReturning aggregate research results to participants: a scoping review of current practices, preferences and challenges
Objectives: To synthesise the current evidence about practices, preferences and challenges related to returning aggregate research results to participants, with implications for public health, health equity and policy development. Design: Scoping review conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews framework. Data sources: Four electronic databases—PubMed (National Library of Medicine), Excerpta Medica Database (Elsevier), Cumulative Index to Nursing and Allied Health Literature (Elton B. Stephens CO(mpany)) and Cochrane Library (Wiley)—were from inception to February 2025. Eligibility criteria: English-language, peer-reviewed articles reporting practices or preferences for returning aggregate clinical research results to participants. Studies reporting only individual/incidental findings, opinion pieces and non-original research were excluded. Data extraction and synthesis: Two reviewers independently screened studies and extracted data. Study quality was assessed using a modified Oxford Centre for Evidence-Based Medicine scale. Thematic synthesis identified patterns in dissemination methods, participant preferences and implementation barriers. Results: Of 272 articles screened, 12 published between 2002 and 2019 were included. They employed cross-sectional, qualitative and mixed-methods design across North America, Europe, Africa and Australia. Half focused on cancer; others addressed malaria, autism, hypothyroidism, HIV prevention and preterm labour. Most (8/12) included patients; others also included researchers, caregivers or mixed stakeholders. Participants were predominantly middle-aged or older, female and well-educated. Only two studies assessed literacy, both using unvalidated self-reports. Eight studies explored preferences without returning results; four implemented disseminations via mailed reports, printed summaries, in-person discussions or digital platforms. Participants favoured mailed letters and face-to-face meetings over online methods. Six themes emerged: receiving results as respect; preference for personalised, timely formats; importance of cultural tailoring; emotional and ethical considerations; institutional barriers; and community engagement as a facilitator. Conclusions: Returning aggregate research results to participants is ethically supported and strongly desired yet rarely practised. Bridging this gap requires clearer policy guidance, institutional support and equity-focused dissemination strategies. Health systems and researchers should adopt scalable, participant-centred approaches to fulfil ethical obligations and strengthen public trust in science.
- 9 November 2024
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 32 (1) , 119-128
Mini-mental status examination phenotyping for Alzheimer’s disease patients using both structured and narrative electronic health record features
Objective: This study aims to automate the prediction of Mini-Mental State Examination (MMSE) scores, a widely adopted standard for cognitive assessment in patients with Alzheimer’s disease, using natural language processing (NLP) and machine learning (ML) on structured and unstructured EHR data. Materials and Methods: We extracted demographic data, diagnoses, medications, and unstructured clinical visit notes from the EHRs. We used Latent Dirichlet Allocation (LDA) for topic modeling and Term-Frequency Inverse Document Frequency (TF-IDF) for n-grams. In addition, we extracted meta-features such as age, ethnicity, and race. Model training and evaluation employed eXtreme Gradient Boosting (XGBoost), Stochastic Gradient Descent Regressor (SGDRegressor), and Multi-Layer Perceptron (MLP). Results: We analyzed 1654 clinical visit notes collected between September 2019 and June 2023 for 1000 Alzheimer’s disease patients. The average MMSE score was 20, with patients averaging 76.4 years old, 54.7% female, and 54.7% identifying as White. The best-performing model (ie, lowest root mean squared error (RMSE)) is MLP, which achieved an RMSE of 5.53 on the validation set using n-grams, indicating superior prediction performance over other models and feature sets. The RMSE on the test set was 5.85. Discussion: This study developed a ML method to predict MMSE scores from unstructured clinical notes, demonstrating the feasibility of utilizing NLP to support cognitive assessment. Future work should focus on refining the model and evaluating its clinical relevance across diverse settings. Conclusion: We contributed a model for automating MMSE estimation using EHR features, potentially transforming cognitive assessment for Alzheimer’s patients and paving the way for more informed clinical decisions and cohort identification.
- 9 November 2024
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 32 (1) , 119-128
Mini-mental status examination phenotyping for Alzheimer’s disease patients using both structured and narrative electronic health record features
Objective: This study aims to automate the prediction of Mini-Mental State Examination (MMSE) scores, a widely adopted standard for cognitive assessment in patients with Alzheimer’s disease, using natural language processing (NLP) and machine learning (ML) on structured and unstructured EHR data. Materials and Methods: We extracted demographic data, diagnoses, medications, and unstructured clinical visit notes from the EHRs. We used Latent Dirichlet Allocation (LDA) for topic modeling and Term-Frequency Inverse Document Frequency (TF-IDF) for n-grams. In addition, we extracted meta-features such as age, ethnicity, and race. Model training and evaluation employed eXtreme Gradient Boosting (XGBoost), Stochastic Gradient Descent Regressor (SGDRegressor), and Multi-Layer Perceptron (MLP). Results: We analyzed 1654 clinical visit notes collected between September 2019 and June 2023 for 1000 Alzheimer’s disease patients. The average MMSE score was 20, with patients averaging 76.4 years old, 54.7% female, and 54.7% identifying as White. The best-performing model (ie, lowest root mean squared error (RMSE)) is MLP, which achieved an RMSE of 5.53 on the validation set using n-grams, indicating superior prediction performance over other models and feature sets. The RMSE on the test set was 5.85. Discussion: This study developed a ML method to predict MMSE scores from unstructured clinical notes, demonstrating the feasibility of utilizing NLP to support cognitive assessment. Future work should focus on refining the model and evaluating its clinical relevance across diverse settings. Conclusion: We contributed a model for automating MMSE estimation using EHR features, potentially transforming cognitive assessment for Alzheimer’s patients and paving the way for more informed clinical decisions and cohort identification.
- Open Access
- 16 October 2024
- Published by Cambridge University Press (CUP) in Journal of Clinical and Translational Science
- Vol. 8 (1) , e167
Clinical researchers’ insights on key data for eligibility screening in clinical studies
Introduction: Clinical research is critical for healthcare advancement, but participant recruitment remains challenging. Clinical research professionals (CRPs; e.g., clinical research coordinator, research assistant) perform eligibility prescreening, ensuring adherence to study criteria while upholding scientific and ethical standards. This study investigates the key information CRP prioritizes during eligibility prescreening, providing insights to optimize data standardization, and recruitment approaches. Methods: We conducted a freelisting survey targeting 150 CRPs from diverse domains (i.e., neurological disorders, rare diseases, and other diseases) where they listed essential information they look for from medical records, participant/caregiver inquiries, and discussions with principal investigators to determine a potential participant’s research eligibility. We calculated the salience scores of listed items using Anthropac, followed by a two-level analytic procedure to classify and thematically categorize the data. Results: The majority of participants were female (81%), identified as White (44%) and as non-Hispanic (64.5%). The first-level analysis universally emphasized age, medication list, and medical history across all domains. The second-level analysis illuminated domain-specific approaches in information retrieval: for instance, history of present illness was notably significant in neurological disorders during participant and principal investigator inquiries, while research participation was distinctly salient in potential participant inquiries within the rare disease domain. Conclusion: This study unveils the intricacies of eligibility prescreening, with both universal and domain-specific methods observed. Variations in data use across domains suggest the need for tailored prescreening in clinical research. Incorporating these insights into CRP training and refining prescreening tools, combined with an ethical, participant-focused approach, can advance eligibility prescreening practices.
- 16 October 2024
- Published by Cambridge University Press (CUP) in Journal of Clinical and Translational Science
- Vol. 8 (1) , e167
Open AccessClinical researchers’ insights on key data for eligibility screening in clinical studies
Introduction: Clinical research is critical for healthcare advancement, but participant recruitment remains challenging. Clinical research professionals (CRPs; e.g., clinical research coordinator, research assistant) perform eligibility prescreening, ensuring adherence to study criteria while upholding scientific and ethical standards. This study investigates the key information CRP prioritizes during eligibility prescreening, providing insights to optimize data standardization, and recruitment approaches. Methods: We conducted a freelisting survey targeting 150 CRPs from diverse domains (i.e., neurological disorders, rare diseases, and other diseases) where they listed essential information they look for from medical records, participant/caregiver inquiries, and discussions with principal investigators to determine a potential participant’s research eligibility. We calculated the salience scores of listed items using Anthropac, followed by a two-level analytic procedure to classify and thematically categorize the data. Results: The majority of participants were female (81%), identified as White (44%) and as non-Hispanic (64.5%). The first-level analysis universally emphasized age, medication list, and medical history across all domains. The second-level analysis illuminated domain-specific approaches in information retrieval: for instance, history of present illness was notably significant in neurological disorders during participant and principal investigator inquiries, while research participation was distinctly salient in potential participant inquiries within the rare disease domain. Conclusion: This study unveils the intricacies of eligibility prescreening, with both universal and domain-specific methods observed. Variations in data use across domains suggest the need for tailored prescreening in clinical research. Incorporating these insights into CRP training and refining prescreening tools, combined with an ethical, participant-focused approach, can advance eligibility prescreening practices.
- 3 June 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 156, 104663
Promoting equity in clinical research: The role of social determinants of health
- 3 June 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 156, 104663
Promoting equity in clinical research: The role of social determinants of health
- 4 May 2024
- Published by Springer Nature in AIDS and Behavior
- Vol. 28 (9) , 2836-2849
Social Marketing Perspective on Participant Recruitment in Informatics-Based Intervention Studies
Effective recruitment strategies are pivotal for informatics-based intervention trials success, particularly for people living with HIV (PLWH), where engagement can be challenging. Although informatics interventions are recognized for improving health outcomes, the effectiveness of their recruitment strategies remains unclear. We investigated the application of a social marketing framework in navigating the nuances of recruitment for informatics-based intervention trials for PLWH by examining participant experiences and perceptions. We used qualitative descriptive methodology to conduct semi-structured interviews with 90 research participants from four informatics-based intervention trials. Directed inductive and deductive content analyses were guided by Howcutt et al.’s social marketing framework on applying the decision-making process to research recruitment. The majority were male (86.7%), living in the Northeast United States (56%), and identified as Black (32%) or White (32%). Most participants (60%) completed the interview remotely. Sixteen subthemes emerged from five themes: motivation, perception, attitude formation, integration, and learning. Findings from our interview data suggest that concepts from Howcutt et al.’s framework informed participants’ decisions to participate in an informatics-based intervention trial. We found that the participants’ perceptions of trust in the research process were integral to the participants across the four trials. However, the recruitment approach and communication medium preferences varied between older and younger age groups. Social marketing framework can provide insight into improving the research recruitment process. Future work should delve into the complex interplay between the type of informatics-based interventions, trust in the research process, and communication preferences, and how these factors collectively influence participants’ willingness to engage.
- 4 May 2024
- Published by Springer Nature in AIDS and Behavior
- Vol. 28 (9) , 2836-2849
Social Marketing Perspective on Participant Recruitment in Informatics-Based Intervention Studies
Effective recruitment strategies are pivotal for informatics-based intervention trials success, particularly for people living with HIV (PLWH), where engagement can be challenging. Although informatics interventions are recognized for improving health outcomes, the effectiveness of their recruitment strategies remains unclear. We investigated the application of a social marketing framework in navigating the nuances of recruitment for informatics-based intervention trials for PLWH by examining participant experiences and perceptions. We used qualitative descriptive methodology to conduct semi-structured interviews with 90 research participants from four informatics-based intervention trials. Directed inductive and deductive content analyses were guided by Howcutt et al.’s social marketing framework on applying the decision-making process to research recruitment. The majority were male (86.7%), living in the Northeast United States (56%), and identified as Black (32%) or White (32%). Most participants (60%) completed the interview remotely. Sixteen subthemes emerged from five themes: motivation, perception, attitude formation, integration, and learning. Findings from our interview data suggest that concepts from Howcutt et al.’s framework informed participants’ decisions to participate in an informatics-based intervention trial. We found that the participants’ perceptions of trust in the research process were integral to the participants across the four trials. However, the recruitment approach and communication medium preferences varied between older and younger age groups. Social marketing framework can provide insight into improving the research recruitment process. Future work should delve into the complex interplay between the type of informatics-based interventions, trust in the research process, and communication preferences, and how these factors collectively influence participants’ willingness to engage.
- 6 March 2024
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 31 (5) , 1062-1073
Sociotechnical feasibility of natural language processing-driven tools in clinical trial eligibility prescreening for Alzheimer’s disease and related dementias
Background: Alzheimer’s disease and related dementias (ADRD) affect over 55 million globally. Current clinical trials suffer from low recruitment rates, a challenge potentially addressable via natural language processing (NLP) technologies for researchers to effectively identify eligible clinical trial participants. Objective: This study investigates the sociotechnical feasibility of NLP-driven tools for ADRD research prescreening and analyzes the tools’ cognitive complexity's effect on usability to identify cognitive support strategies. Methods: A randomized experiment was conducted with 60 clinical research staff using three prescreening tools (Criteria2Query, Informatics for Integrating Biology and the Bedside [i2b2], and Leaf). Cognitive task analysis was employed to analyze the usability of each tool using the Health Information Technology Usability Evaluation Scale. Data analysis involved calculating descriptive statistics, interrater agreement via intraclass correlation coefficient, cognitive complexity, and Generalized Estimating Equations models. Results: Leaf scored highest for usability followed by Criteria2Query and i2b2. Cognitive complexity was found to be affected by age, computer literacy, and number of criteria, but was not significantly associated with usability. Discussion: Adopting NLP for ADRD prescreening demands careful task delegation, comprehensive training, precise translation of eligibility criteria, and increased research accessibility. The study highlights the relevance of these factors in enhancing NLP-driven tools’ usability and efficacy in clinical research prescreening. Conclusion: User-modifiable NLP-driven prescreening tools were favorably received, with system type, evaluation sequence, and user's computer literacy influencing usability more than cognitive complexity. The study emphasizes NLP's potential in improving recruitment for clinical trials, endorsing a mixed-methods approach for future system evaluation and enhancements.
- 6 March 2024
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 31 (5) , 1062-1073
Sociotechnical feasibility of natural language processing-driven tools in clinical trial eligibility prescreening for Alzheimer’s disease and related dementias
Background: Alzheimer’s disease and related dementias (ADRD) affect over 55 million globally. Current clinical trials suffer from low recruitment rates, a challenge potentially addressable via natural language processing (NLP) technologies for researchers to effectively identify eligible clinical trial participants. Objective: This study investigates the sociotechnical feasibility of NLP-driven tools for ADRD research prescreening and analyzes the tools’ cognitive complexity's effect on usability to identify cognitive support strategies. Methods: A randomized experiment was conducted with 60 clinical research staff using three prescreening tools (Criteria2Query, Informatics for Integrating Biology and the Bedside [i2b2], and Leaf). Cognitive task analysis was employed to analyze the usability of each tool using the Health Information Technology Usability Evaluation Scale. Data analysis involved calculating descriptive statistics, interrater agreement via intraclass correlation coefficient, cognitive complexity, and Generalized Estimating Equations models. Results: Leaf scored highest for usability followed by Criteria2Query and i2b2. Cognitive complexity was found to be affected by age, computer literacy, and number of criteria, but was not significantly associated with usability. Discussion: Adopting NLP for ADRD prescreening demands careful task delegation, comprehensive training, precise translation of eligibility criteria, and increased research accessibility. The study highlights the relevance of these factors in enhancing NLP-driven tools’ usability and efficacy in clinical research prescreening. Conclusion: User-modifiable NLP-driven prescreening tools were favorably received, with system type, evaluation sequence, and user's computer literacy influencing usability more than cognitive complexity. The study emphasizes NLP's potential in improving recruitment for clinical trials, endorsing a mixed-methods approach for future system evaluation and enhancements.
- Open Access
- 4 January 2024
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 7 (1) , ooae021
Retrieval augmented scientific claim verification
To automate scientific claim verification using PubMed abstracts. We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER’s performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021. In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively. CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.
- 4 January 2024
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 7 (1) , ooae021
Open AccessRetrieval augmented scientific claim verification
To automate scientific claim verification using PubMed abstracts. We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER’s performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021. In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively. CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.
- Top Cited PapersOpen Access
- 24 August 2023
- Published by Springer Nature in npj Digital Medicine
- Vol. 6 (1) , 1-8
Evaluating large language models on medical evidence summarization
Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.
- 24 August 2023
- Published by Springer Nature in npj Digital Medicine
- Vol. 6 (1) , 1-8
Top Cited PapersOpen AccessEvaluating large language models on medical evidence summarization
Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.
- Open Access
- 23 August 2023
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 30 (12) , 1895-1903
The suitability of UMLS and SNOMED-CT for encoding outcome concepts
Outcomes are important clinical study information. Despite progress in automated extraction of PICO (Population, Intervention, Comparison, and Outcome) entities from PubMed, rarely are these entities encoded by standard terminology to achieve semantic interoperability. This study aims to evaluate the suitability of the Unified Medical Language System (UMLS) and SNOMED-CT in encoding outcome concepts in randomized controlled trial (RCT) abstracts. We iteratively developed and validated an outcome annotation guideline and manually annotated clinically significant outcome entities in the Results and Conclusions sections of 500 randomly selected RCT abstracts on PubMed. The extracted outcomes were fully, partially, or not mapped to the UMLS via MetaMap based on established heuristics. Manual UMLS browser search was performed for select unmapped outcome entities to further differentiate between UMLS and MetaMap errors. Only 44% of 2617 outcome concepts were fully covered in the UMLS, among which 67% were complex concepts that required the combination of 2 or more UMLS concepts to represent them. SNOMED-CT was present as a source in 61% of the fully mapped outcomes. Domains such as Metabolism and Nutrition, and Infections and Infectious Diseases need expanded outcome concept coverage in the UMLS and MetaMap. Future work is warranted to similarly assess the terminology coverage for P, I, C entities. Computational representation of clinical outcomes is important for clinical evidence extraction and appraisal and yet faces challenges from the inherent complexity and lack of coverage of these concepts in UMLS and SNOMED-CT, as demonstrated in this study.
- 23 August 2023
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 30 (12) , 1895-1903
Open AccessThe suitability of UMLS and SNOMED-CT for encoding outcome concepts
Outcomes are important clinical study information. Despite progress in automated extraction of PICO (Population, Intervention, Comparison, and Outcome) entities from PubMed, rarely are these entities encoded by standard terminology to achieve semantic interoperability. This study aims to evaluate the suitability of the Unified Medical Language System (UMLS) and SNOMED-CT in encoding outcome concepts in randomized controlled trial (RCT) abstracts. We iteratively developed and validated an outcome annotation guideline and manually annotated clinically significant outcome entities in the Results and Conclusions sections of 500 randomly selected RCT abstracts on PubMed. The extracted outcomes were fully, partially, or not mapped to the UMLS via MetaMap based on established heuristics. Manual UMLS browser search was performed for select unmapped outcome entities to further differentiate between UMLS and MetaMap errors. Only 44% of 2617 outcome concepts were fully covered in the UMLS, among which 67% were complex concepts that required the combination of 2 or more UMLS concepts to represent them. SNOMED-CT was present as a source in 61% of the fully mapped outcomes. Domains such as Metabolism and Nutrition, and Infections and Infectious Diseases need expanded outcome concept coverage in the UMLS and MetaMap. Future work is warranted to similarly assess the terminology coverage for P, I, C entities. Computational representation of clinical outcomes is important for clinical evidence extraction and appraisal and yet faces challenges from the inherent complexity and lack of coverage of these concepts in UMLS and SNOMED-CT, as demonstrated in this study.
- 2 May 2023
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 142, 104375
A data-driven approach to optimizing clinical study eligibility criteria
- 2 May 2023
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 142, 104375
A data-driven approach to optimizing clinical study eligibility criteria