TKTaneya Y. Koonce

Taneya Y. Koonce

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Taneya Y. Koonce has a total of 80 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Evaluating Eight Retrieval-Augmented Generation (RAG) Large Language Models’ Responses to Clinical Questions: A Comparative Study

    by Poppy A. Krump, Mallory N. Blasingame, Taneya Y. Koonce, Annette M. Williams, Jing Su, Nunzia B. Giuse

    Background: Large language models (LLMs) that use retrieval-augmented generation (RAG) are increasingly used to answer clinical questions, although the evaluation of these systems remains limited. Building on previous studies conducted by our team, this case report aimed to improve upon this knowledge gap by applying a reusable methodology to compare the performance of eight LLMs that utilize RAG techniques for evidence synthesis. Case Presentation: Eight commercially available RAG LLM tools (OpenEvidence, Undermind, Consensus, SciSpace, Elicit, MediSearch, EvidenceHunt, and Scite) were evaluated using twelve ChatGPT-generated clinical questions on the topics of treatment, etiology, and prognosis. To enable comparison, we prompted ChatGPT to identify all key unique medical concepts from the full set of LLM responses to each question. Concepts were categorized as critical (“must-have”) or non-critical (“nice-to-have”) for answering the clinical question. Experienced information scientists were consulted at each step for their expertise. Descriptive statistics and Kruskal-Wallis tests were used to compare performance across tools and question categories. No significant differences were found among the eight RAG LLMs in their coverage of “must-have” (p=0.95) or “nice-to-have” (p=0.16) key unique medical concepts, and no single tool consistently captured all identified concepts. Conclusions: These findings suggest that RAG LLMs may be supplementary tools for evidence retrieval and synthesis but cannot, at this time, fully replace comprehensive expert review of the medical literature. The evaluation framework presented here may be a useful model for future comparative assessments of rapidly evolving AI evidence synthesis tools.

  • Open Access

    Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries

    by Mallory N. Blasingame, Taneya Y. Koonce, Annette M. Williams, Jing Su, Dario A. Giuse, Poppy A. Krump, Nunzia B. Giuse

    Objective: To compare answers to clinical questions between five publicly available large language model (LLM) chatbots and information scientists. Methods: LLMs were prompted to provide 45 PICO (patient, intervention, comparison, outcome) questions addressing treatment, prognosis, and etiology. Each question was answered by a medical information scientist and submitted to five LLM tools: ChatGPT, Gemini, Copilot, DeepSeek, and Grok-3. Key elements from the answers provided were used by pairs of information scientists to label each LLM answer as in Total Alignment, Partial Alignment, or No Alignment with the information scientist. The Partial Alignment answers were also analyzed for the inclusion of additional information. Results: The entire LLM set of answers, 225 in total, were assessed as being in Total Alignment 20.9% of the time (n=47), in Partial Alignment 78.7% of the time (n=177), and in No Alignment 0.4% of the time (n=1). Kruskal-Wallis testing found no significant performance difference in alignment ratings between the five chatbots (p=0.46). An analysis of the partially aligned answers found a significant difference in the number of additional elements provided by the information scientists versus the chatbots per Wilcoxon-Rank Sum testing (p=0.02). Discussion: Five chatbots did not differ significantly in their alignment with information scientists’ evidence summaries. The analysis of partially aligned answers found both chatbots and information scientists included additional information, with information scientists doing so significantly more often. An important next step will be to assess the additional information, both from the chatbots and the information scientists for validity and relevance.


  • Open Access

    Comparing Five Generative AI Chatbots’ Answers to LLM-Generated Clinical Questions with Medical Information Scientists’ Evidence Summaries

    by Mallory N. Blasingame, Taneya Y. Koonce, Annette M. Williams, Jing Su, Dario A. Giuse, Poppy A. Krump, Nunzia B. Giuse

    Objective: To compare answers to clinical questions between five publicly available large language model (LLM) chatbots and information scientists. Methods: LLMs were prompted to provide 45 PICO (patient, intervention, comparison, outcome) questions addressing treatment, prognosis, and etiology. Each question was answered by a medical information scientist and submitted to five LLM tools: ChatGPT, Gemini, Copilot, DeepSeek, and Grok-3. Key elements from the answers provided were used by pairs of information scientists to label each LLM answer as in Total Alignment, Partial Alignment, or No Alignment with the information scientist. The Partial Alignment answers were also analyzed for the inclusion of additional information. Results: The entire LLM set of answers, 225 in total, were assessed as being in Total Alignment 20.9% of the time (n=47), in Partial Alignment 78.7% of the time (n=177), and in No Alignment 0.4% of the time (n=1). Kruskal-Wallis testing found no significant performance difference in alignment ratings between the five chatbots ( p =0.46). An analysis of the partially aligned answers found a significant difference in the number of additional elements provided by the information scientists versus the chatbots per Wilcoxon-Rank Sum testing ( p =0.02). Discussion: Five chatbots did not differ significantly in their alignment with information scientists’ evidence summaries. The analysis of partially aligned answers found both chatbots and information scientists included additional information, with information scientists doing so significantly more often. An important next step will be to assess the additional information both from the chatbots and the information scientists for validity and relevance.

  • Open Access

    Development of an open access systematic review instructional video series accessible through the SPI-HubTM website

    by Sheila Kusnoor, Annette Williams, Taneya Koonce, Poppy Krump, Lori Harding, Jerry Zhao, John Clark, Nunzia Giuse

    Given the key role of systematic reviews in informing clinical decision making and guidelines, it is important for individuals to have equitable access to quality instructional materials on how to design, conduct, report, and evaluate systematic reviews. In response to this need, Vanderbilt University Medical Center’s Center for Knowledge Management (CKM) created an open-access systematic review instructional video series. The educational content was created by experienced CKM information scientists, who worked together to adapt an internal training series that they had developed into a format that could be widely shared with the public. Brief videos, averaging 10 minutes in length, were created addressing essential concepts related to systematic reviews, including distinguishing between literature review types, understanding reasons for conducting a systematic review, designing a systematic review protocol, steps in conducting a systematic review, web-based tools to aid with the systematic review process, publishing a systematic review, and critically evaluating systematic reviews. Quiz questions were developed for each instructional video to allow learners to check their understanding of the material. The systematic review instructional video series launched on CKM’s Scholarly Publishing Information Hub (SPI-HubTM) website in Fall 2023. From January through August 2024, there were 1,662 international accesses to the SPI-HubTM systematic review website, representing 41 countries. Initial feedback, while primarily anecdotal, has been positive. By adapting its internal systematic review training into an online video series format suitable for asynchronous instruction, CKM has been able to widely disseminate its educational materials.

  • Open Access

    Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries

    by Mallory N. Blasingame, Taneya Y. Koonce, Annette M. Williams, Dario A. Giuse, Jing Su, Poppy A. Krump, Nunzia Bettinsoli Giuse

    Objective: This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians’ gold-standard evidence syntheses. Methods: Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat. Results: Of the 216 evaluated questions, aiChat’s response was assessed as “correct” for 180 (83.3%) questions, “partially correct” for 35 (16.2%) questions, and “incorrect” for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.73). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated. Conclusions: Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians’ workflow.


  • Open Access

    Evaluating a Large Language Model’s Ability to Answer Clinicians’ Requests for Evidence Summaries

    by Mallory N. Blasingame, Taneya Y. Koonce, Annette M. Williams, Dario A. Giuse, Jing Su, Poppy A. Krump, Nunzia Bettinsoli Giuse

    Objective This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians’ gold-standard evidence syntheses. Methods Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally-managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat. Results Of the 216 evaluated questions, aiChat’s response was assessed as “correct” for 180 (83.3%) questions, “partially correct” for 35 (16.2%) questions, and “incorrect” for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.39). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated. Conclusions Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians’ workflow.


  • Open Access

    Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition

    by Taneya Y Koonce, Dario A Giuse, Annette M Williams, Mallory N Blasingame, Poppy A Krump, Jing Su, Nunzia B Giuse

    Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC’s electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies.


  • Open Access

    Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition (Preprint)

    by Taneya Y Koonce, Dario A Giuse, Annette M Williams, Mallory N Blasingame, Poppy A Krump, Jing Su, Nunzia B Giuse

    UNSTRUCTURED Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC’s electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies.

  • Knowledgebase strategies to aid interpretation of clinical correlation research

    by William W Stead, Adam Lewis, Nunzia B Giuse, Taneya Y Koonce, Lisa Bastarache

    Objective: Knowledgebases are needed to clarify correlations observed in real-world electronic health record (EHR) data. We posit design principles, present a unifying framework, and report a test of concept. Materials and Methods: We structured a knowledge framework along 3 axes: condition of interest, knowledge source, and taxonomy. In our test of concept, we used hypertension as our condition of interest, literature and VanderbiltDDx knowledgebase as sources, and phecodes as our taxonomy. In a cohort of 832 566 deidentified EHRs, we modeled blood pressure and heart rate by sex and age, classified individuals by hypertensive status, and ran a Phenome-wide Association Study (PheWAS) for hypertension. We compared the correlations from PheWAS to the associations in our knowledgebase. Results: We produced PhecodeKbHtn: a knowledgebase comprising 167 hypertension-associated diseases, 15 of which were also negatively associated with blood pressure (pos+neg). Our hypertension PheWAS included 1914 phecodes, 129 of which were in the PhecodeKbHtn. Among the PheWAS association results, phecodes that were in PhecodeKbHtn had larger effect sizes compared with those phecodes not in the knowledgebase. Discussion: Each source contributed unique and additive associations. Models of blood pressure and heart rate by age and sex were consistent with prior cohort studies. All but 4 PheWAS positive and negative correlations for phecodes in PhecodeKbHtn may be explained by knowledgebase associations, hypertensive cardiac complications, or causes of hypertension independently associated with hypotension. Conclusion: It is feasible to assemble a knowledgebase that is compatible with EHR data to aid interpretation of clinical correlation research.


  • Open Access

    Systematic review of international studies evaluating MDRD and CKD-EPI estimated glomerular filtration rate (eGFR) equations in Black adults

    by Ebele M. Umeukeje, Taneya Y. Koonce, Sheila V. Kusnoor, Ifeoma I. Ulasi, Sophia Kostelanetz, Annette M. Williams, Mallory N. Blasingame, Marcia I. Epelbaum, Dario A. Giuse, Annie N. Apple, et al.

    Use of race adjustment in estimating glomerular filtration rate (eGFR) has been challenged given concerns that it may negatively impact the clinical care of Black patients, as it results in Black patients being systematically assigned higher eGFR values than non-Black patients. We conducted a systematic review to assess how well eGFR, with and without race adjustment, estimates measured GFR (mGFR) in Black adults globally. A search across multiple databases for articles published from 1999 to May 2021 that compared eGFR to mGFR and reported outcomes by Black race was performed. We included studies that assessed eGFR using the Modification of Diet in Renal Disease (MDRD) and Chronic Kidney Disease Epidemiology Collaboration (CKD-EPICr) creatinine equations. Risk of study bias and applicability were assessed with the QUality Assessment of Diagnostic Accuracy Studies-2. Of 13,167 citations identified, 12 met the data synthesis criteria (unique patient cohorts in which eGFR was compared to mGFR with and without race adjustment). The studies included patients with and without kidney disease from Africa (n = 6), the United States (n = 3), Europe (n = 2), and Brazil (n = 1). Of 11 CKD-EPI equation studies, all assessed bias, 8 assessed accuracy, 6 assessed precision, and 5 assessed correlation/concordance. Of 7 MDRD equation studies, all assessed bias, 6 assessed accuracy, 5 assessed precision, and 3 assessed correlation/concordance. The majority of studies found that removal of race adjustment improved bias, accuracy, and precision of eGFR equations for Black adults. Risk of study bias was often unclear, but applicability concerns were low. Our systematic review supports the need for future studies to be conducted in diverse populations to assess the possibility of alternative approaches for estimating GFR. This study additionally provides systematic-level evidence for the American Society of Nephrology—National Kidney Foundation Task Force efforts to pursue other options for GFR estimation.