Kevin Wu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Kevin Wu has a total of 45 co-authors. Here are the top co-authors by number of shared publications:
- James Zou
Department of Biomedical Data Science, Stanford University, Stanford, USA
9 shared publications - Howard Y. Chang
Departments of Dermatology, Genetics, and Pathology, Stanford University School of Medicine, Stanford, USA
5 shared publications Angela Zhang
Department of Genetics, Stanford University, Stanford, USA
2 shared publicationsKathryn E. Yost
Center for Personal and Dynamic Regulomes, Stanford University School of Medicine, Stanford, CA 94305
2 shared publicationsKevin Wei
Keck Medicine of USC, Los Angeles, USA
2 shared publicationsSith Riantawan
Keck Medicine of USC, Los Angeles, USA
2 shared publications
Latest Publications
- Top Cited PapersOpen Access
- 16 April 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-10
An automated framework for assessing how well LLMs cite relevant medical references
As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references.
- 16 April 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-10
Top Cited PapersOpen AccessAn automated framework for assessing how well LLMs cite relevant medical references
As large language models (LLMs) are increasingly used to address health-related queries, it is crucial that they support their conclusions with credible references. While models can cite sources, the extent to which these support claims remains unclear. To address this gap, we introduce SourceCheckup, an automated agent-based pipeline that evaluates the relevance and supportiveness of sources in LLM responses. We evaluate seven popular LLMs on a dataset of 800 questions and 58,000 pairs of statements and sources on data that represent common medical queries. Our findings reveal that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite. Even for GPT-4o with Web Search, approximately 30% of individual statements are unsupported, and nearly half of its responses are not fully supported. Independent assessments by doctors further validate these results. Our research underscores significant limitations in current LLMs to produce trustworthy medical references.
- Open Access
ProteinCLIP: enhancing protein language models with natural language
Language models have enabled a new era of biological sequence modeling. However, extracting meaningful sequence-level embeddings from these models remains challenging. In this work, we introduce ProteinCLIP, which applies contrastive learning between a protein’s amino acid sequence and curated text describing its function. ProteinCLIP thus learns to take a pre-trained protein language model’s sequence embedding and refines it produce a function-centric embedding. We show that this embedding space yields sequence representations that enable state-of-the-art performance across a variety of important yet challenging tasks in the study of proteins – from predicting protein protein interactions to accurately detecting homologous proteins despite low sequence similarity. More broadly, ProteinCLIP demonstrates the effectiveness of multi-modal learning in biological contexts, and how such strategies can help isolate key signals from large models and further improve their utility.
Open AccessProteinCLIP: enhancing protein language models with natural language
Language models have enabled a new era of biological sequence modeling. However, extracting meaningful sequence-level embeddings from these models remains challenging. In this work, we introduce ProteinCLIP, which applies contrastive learning between a protein’s amino acid sequence and curated text describing its function. ProteinCLIP thus learns to take a pre-trained protein language model’s sequence embedding and refines it produce a function-centric embedding. We show that this embedding space yields sequence representations that enable state-of-the-art performance across a variety of important yet challenging tasks in the study of proteins – from predicting protein protein interactions to accurately detecting homologous proteins despite low sequence similarity. More broadly, ProteinCLIP demonstrates the effectiveness of multi-modal learning in biological contexts, and how such strategies can help isolate key signals from large models and further improve their utility.
- Open Access
How well do LLMs cite relevant medical references? An evaluation framework and analyses
Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.
Open AccessHow well do LLMs cite relevant medical references? An evaluation framework and analyses
Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.
- Open Access
- 15 June 2022
- Published by American Association for Cancer Research (AACR) in Cancer Research
- Vol. 82 (12_Supplem) , 635
Abstract 635: Concordance between dogs and humans: The use of AI in evaluating clinical cancer genomic datasets
Spontaneous tumors in dogs are highly similar to human cancers histologically, genetically, molecularly and clinically. In this presentation we explore similarities found across species when associating cancer biomarkers and outcomes. Our findings validate concordance across the cancer journeys. This is important to deepen the field of comparative oncology and affirming spontaneous cancer in dogs as an ideal ‘model’ for precision oncology, building upon recent work in tumor biology concordance. Our machine learning analysis prognosis and predictive effect of genomic alterations in 1303 client-owned tumor-bearing dogs identified TP53 and PTPRD as prognostic markers for poor survival in dogs with an overall survival hazard ratio (OS HR) of 1.732, 1.914, respectively (PCitation Format: Kevin Wu, Lucas Rodrigues, Gerald Post, Garrett Harvey, Aubrey Miller, Lindsay Lambert, Christina Lopes, Benjamin Lewis, James Zou. Concordance between dogs and humans: The use of AI in evaluating clinical cancer genomic datasets [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 635.
- 15 June 2022
- Published by American Association for Cancer Research (AACR) in Cancer Research
- Vol. 82 (12_Supplem) , 635
Open AccessAbstract 635: Concordance between dogs and humans: The use of AI in evaluating clinical cancer genomic datasets
Spontaneous tumors in dogs are highly similar to human cancers histologically, genetically, molecularly and clinically. In this presentation we explore similarities found across species when associating cancer biomarkers and outcomes. Our findings validate concordance across the cancer journeys. This is important to deepen the field of comparative oncology and affirming spontaneous cancer in dogs as an ideal ‘model’ for precision oncology, building upon recent work in tumor biology concordance. Our machine learning analysis prognosis and predictive effect of genomic alterations in 1303 client-owned tumor-bearing dogs identified TP53 and PTPRD as prognostic markers for poor survival in dogs with an overall survival hazard ratio (OS HR) of 1.732, 1.914, respectively (PCitation Format: Kevin Wu, Lucas Rodrigues, Gerald Post, Garrett Harvey, Aubrey Miller, Lindsay Lambert, Christina Lopes, Benjamin Lewis, James Zou. Concordance between dogs and humans: The use of AI in evaluating clinical cancer genomic datasets [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 635.
- 21 April 2022
- Published by Springer Nature in The AAPS Journal
- Vol. 24 (3) , 1-9
Machine Learning Prediction of Clinical Trial Operational Efficiency
Clinical trials are the gatekeepers and bottlenecks of progress in medicine. In recent years, they have become increasingly complex and expensive, driven b
- 21 April 2022
- Published by Springer Nature in The AAPS Journal
- Vol. 24 (3) , 1-9
Machine Learning Prediction of Clinical Trial Operational Efficiency
Clinical trials are the gatekeepers and bottlenecks of progress in medicine. In recent years, they have become increasingly complex and expensive, driven b
- Open Access
TCR-BERT: learning the grammar of T-cell receptors for flexible antigen-xbinding analyses
The T-cell receptor (TCR) allows T-cells to recognize and respond to antigens presented by infected and diseased cells. However, due to TCRs’ staggering diversity and the complex binding dynamics underlying TCR antigen recognition, it is challenging to predict which antigens a given TCR may bind to. Here, we present TCR-BERT, a deep learning model that applies self-supervised transfer learning to this problem. TCR-BERT leverages unlabeled TCR sequences to learn a general, versatile representation of TCR sequences, enabling numerous downstream applications. We demonstrate that TCR-BERT can be used to build state-of-the-art TCR-antigen binding predictors with improved generalizability compared to prior methods. TCR-BERT simultaneously facilitates clustering sequences likely to share antigen specificities. It also facilitates computational approaches to challenging, unsolved problems such as designing novel TCR sequences with engineered binding affinities. Importantly, TCR-BERT enables all these advances by focusing on residues with known biological significance. TCR-BERT can be a useful tool for T-cell scientists, enabling greater understanding and more diverse applications, and provides a conceptual framework for leveraging unlabeled data to improve machine learning on biological sequences.
Open AccessTCR-BERT: learning the grammar of T-cell receptors for flexible antigen-xbinding analyses
The T-cell receptor (TCR) allows T-cells to recognize and respond to antigens presented by infected and diseased cells. However, due to TCRs’ staggering diversity and the complex binding dynamics underlying TCR antigen recognition, it is challenging to predict which antigens a given TCR may bind to. Here, we present TCR-BERT, a deep learning model that applies self-supervised transfer learning to this problem. TCR-BERT leverages unlabeled TCR sequences to learn a general, versatile representation of TCR sequences, enabling numerous downstream applications. We demonstrate that TCR-BERT can be used to build state-of-the-art TCR-antigen binding predictors with improved generalizability compared to prior methods. TCR-BERT simultaneously facilitates clustering sequences likely to share antigen specificities. It also facilitates computational approaches to challenging, unsolved problems such as designing novel TCR sequences with engineered binding affinities. Importantly, TCR-BERT enables all these advances by focusing on residues with known biological significance. TCR-BERT can be a useful tool for T-cell scientists, enabling greater understanding and more diverse applications, and provides a conceptual framework for leveraging unlabeled data to improve machine learning on biological sequences.
- Open Access
- 7 April 2021
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 118 (15)
BABEL enables cross-modality translation between multiomic profiles at single-cell resolution
Significance: Simultaneous measurement of the DNA, RNA, and proteins of single cells can lead to important new insights but is experimentally challenging. This work introduces a deep learning algorithm that flexibly translates between chromatin, RNA, and protein profiles in single cells. This makes it possible to computationally synthesize matched multiomic measurements when only one modality is experimentally available. This algorithm complements experimental advances to efficiently achieve single-cell multiomic insight.
- 7 April 2021
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 118 (15)
Open AccessBABEL enables cross-modality translation between multiomic profiles at single-cell resolution
Significance: Simultaneous measurement of the DNA, RNA, and proteins of single cells can lead to important new insights but is experimentally challenging. This work introduces a deep learning algorithm that flexibly translates between chromatin, RNA, and protein profiles in single cells. This makes it possible to computationally synthesize matched multiomic measurements when only one modality is experimentally available. This algorithm complements experimental advances to efficiently achieve single-cell multiomic insight.
- Open Access
BABEL enables cross-modality translation between multi-omic profiles at single-cell resolution
Simultaneous profiling of multi-omic modalities within a single cell is a grand challenge for single-cell biology. While there have been impressive technical innovations demonstrating feasibility – for example generating paired measurements of scRNA-seq and scATAC-seq – wide-spread application of joint profiling is challenging due to the experimental complexity, noise, and cost. Here we introduce BABEL, a deep learning method that translates between the transcriptome and chromatin profiles of a single cell. Leveraging a novel interoperable neural network model, BABEL can generate scRNA-seq directly from a cell’s scATAC-seq, and vice versa. This makes it possible to computationally synthesize paired multi-omic measurements when only one modality is experimentally available. Across several paired scRNA-seq and scATAC-seq datasets in human and mouse, we validate that BABEL accurately translates between these modalities for individual cells. BABEL also generalizes well to new biological contexts not seen during training. For example, starting from scATAC-seq of patient derived basal cell carcinoma (BCC), BABEL generated scRNA-seq that enabled fine-grained classification of complex cell states, despite having never seen BCC data. These predictions are comparable to analyses of the experimental BCC scRNA-seq data. We further show that BABEL can incorporate additional single-cell data modalities, such as CITE-seq, thus enabling translation across chromatin, RNA, and protein. BABEL offers a powerful approach for data exploration and hypothesis generation.
Open AccessBABEL enables cross-modality translation between multi-omic profiles at single-cell resolution
Simultaneous profiling of multi-omic modalities within a single cell is a grand challenge for single-cell biology. While there have been impressive technical innovations demonstrating feasibility – for example generating paired measurements of scRNA-seq and scATAC-seq – wide-spread application of joint profiling is challenging due to the experimental complexity, noise, and cost. Here we introduce BABEL, a deep learning method that translates between the transcriptome and chromatin profiles of a single cell. Leveraging a novel interoperable neural network model, BABEL can generate scRNA-seq directly from a cell’s scATAC-seq, and vice versa. This makes it possible to computationally synthesize paired multi-omic measurements when only one modality is experimentally available. Across several paired scRNA-seq and scATAC-seq datasets in human and mouse, we validate that BABEL accurately translates between these modalities for individual cells. BABEL also generalizes well to new biological contexts not seen during training. For example, starting from scATAC-seq of patient derived basal cell carcinoma (BCC), BABEL generated scRNA-seq that enabled fine-grained classification of complex cell states, despite having never seen BCC data. These predictions are comparable to analyses of the experimental BCC scRNA-seq data. We further show that BABEL can incorporate additional single-cell data modalities, such as CITE-seq, thus enabling translation across chromatin, RNA, and protein. BABEL offers a powerful approach for data exploration and hypothesis generation.
- Open Access
RNA-GPS Predicts SARS-CoV-2 RNA Localization to Host Mitochondria and Nucleolus
The SARS-CoV-2 coronavirus is driving a global pandemic, but its biological mechanisms are less well understood. SARS-CoV-2 is an RNA virus whose multiple genomic and sub-genomic RNA (sgRNA) transcripts hijack the host cell’s machinery, located across distinct cytotopic locations. Subcellular localization of its viral RNA could play important roles in viral replication and host antiviral immune response. Here we perform computational modeling of SARS-CoV-2 viral RNA localization across eight subcellular neighborhoods. We compare hundreds of SARS-CoV-2 genomes to the human transcriptome and other coronaviruses and perform systematic sub-sequence analyses to identify the responsible signals. Using state-of-the-art machine learning models, we predict that the SARS-CoV-2 RNA genome and all sgRNAs are enriched in the host mitochondrial matrix and nucleolus. The 5’ and 3’ viral untranslated regions possess the strongest and most distinct localization signals. We discuss the mitochondrial localization signal in relation to the formation of double-membrane vesicles, a critical stage in the coronavirus life cycle. Our computational analysis serves as a hypothesis generation tool to suggest models for SARS-CoV-2 biology and inform experimental efforts to combat the virus.
Open AccessRNA-GPS Predicts SARS-CoV-2 RNA Localization to Host Mitochondria and Nucleolus
The SARS-CoV-2 coronavirus is driving a global pandemic, but its biological mechanisms are less well understood. SARS-CoV-2 is an RNA virus whose multiple genomic and sub-genomic RNA (sgRNA) transcripts hijack the host cell’s machinery, located across distinct cytotopic locations. Subcellular localization of its viral RNA could play important roles in viral replication and host antiviral immune response. Here we perform computational modeling of SARS-CoV-2 viral RNA localization across eight subcellular neighborhoods. We compare hundreds of SARS-CoV-2 genomes to the human transcriptome and other coronaviruses and perform systematic sub-sequence analyses to identify the responsible signals. Using state-of-the-art machine learning models, we predict that the SARS-CoV-2 RNA genome and all sgRNAs are enriched in the host mitochondrial matrix and nucleolus. The 5’ and 3’ viral untranslated regions possess the strongest and most distinct localization signals. We discuss the mitochondrial localization signal in relation to the formation of double-membrane vesicles, a critical stage in the coronavirus life cycle. Our computational analysis serves as a hypothesis generation tool to suggest models for SARS-CoV-2 biology and inform experimental efforts to combat the virus.
- Open Access
- 27 March 2020
- Published by Cold Spring Harbor Laboratory in RNA
- Vol. 26 (7) , 851-865
RNA-GPS predicts high-resolution RNA subcellular localization and highlights the role of splicing
Subcellular localization is essential to RNA biogenesis, processing, and function across the gene expression life cycle. However, the specific nucleotide sequence motifs that direct RNA localization are incompletely understood. Fortunately, new sequencing technologies have provided transcriptome-wide atlases of RNA localization, creating an opportunity to leverage computational modeling. Here we present RNA-GPS, a new machine learning model that uses nucleotide-level features to predict RNA localization across eight different subcellular locations—the first to provide such a wide range of predictions. RNA-GPS's design enables high-throughput sequence ablation and feature importance analyses to probe the sequence motifs that drive localization prediction. We find localization informative motifs to be concentrated on 3′-UTRs and scattered along the coding sequence, and motifs related to splicing to be important drivers of predicted localization, even for cytotopic distinctions for membraneless bodies within the nucleus or for organelles within the cytoplasm. Overall, our results suggest transcript splicing is one of many elements influencing RNA subcellular localization.
- 27 March 2020
- Published by Cold Spring Harbor Laboratory in RNA
- Vol. 26 (7) , 851-865
Open AccessRNA-GPS predicts high-resolution RNA subcellular localization and highlights the role of splicing
Subcellular localization is essential to RNA biogenesis, processing, and function across the gene expression life cycle. However, the specific nucleotide sequence motifs that direct RNA localization are incompletely understood. Fortunately, new sequencing technologies have provided transcriptome-wide atlases of RNA localization, creating an opportunity to leverage computational modeling. Here we present RNA-GPS, a new machine learning model that uses nucleotide-level features to predict RNA localization across eight different subcellular locations—the first to provide such a wide range of predictions. RNA-GPS's design enables high-throughput sequence ablation and feature importance analyses to probe the sequence motifs that drive localization prediction. We find localization informative motifs to be concentrated on 3′-UTRs and scattered along the coding sequence, and motifs related to splicing to be important drivers of predicted localization, even for cytotopic distinctions for membraneless bodies within the nucleus or for organelles within the cytoplasm. Overall, our results suggest transcript splicing is one of many elements influencing RNA subcellular localization.