STonKGs: A Sophisticated Transformer Trained on Biomedical Text and Knowledge Graphs
Authors:
- Helena BalabinHBHelena Balabin
- Charles Tapley HoytCHCharles Tapley Hoyt
- Colin BirkenbihlCBColin Birkenbihl
- Benjamin M GyoriBGBenjamin M Gyori
- John BachmanJBJohn Bachman
- Alpha Tom KodamullilAKAlpha Tom Kodamullil
- Paul G PlögerPPPaul G Plöger
- Martin Hofmann-ApitiusMHMartin Hofmann-Apitius
- Daniel Domingo-FernándezDDDaniel Domingo-Fernández
Preprint
Abstract
The majority of biomedical knowledge is stored in structured databases or as unstructured text in scientific publications. This vast amount of information has led to numerous machine learning-based biological applications using either text through natural language processing (NLP) or structured data through knowledge graph embedding models (KGEMs). However, representations based on a single modality are inherently limited. To generate better representations of biological knowledge, we propose STonKGs, a Sophisticated Transformer trained on biomedical text and Knowledge Graphs. This multimodal Transformer uses combined input sequences of structured information from KGs and unstructured text data from biomedical literature to learn joint representations. First, we pre-trained STonKGs on a knowledge base assembled by the Integrated Network and Dynamical Reasoning Assembler (INDRA) consisting of millions of text-triple pairs extracted from biomedical literature by multiple NLP systems. Then, we benchmarked STonKGs against two baseline models trained on either one of the modalities (i.e., text or KG) across eight different classification tasks, each corresponding to a different biological application. Our results demonstrate that STonKGs outperforms both baselines, especially on the more challenging tasks with respect to the number of classes, improving upon the F1-score of the best baseline by up to 0.083. Additionally, our pre-trained model as well as the model architecture can be adapted to various other transfer learning applications. Finally, the source code and pre-trained STonKGs models are available at https://github.com/stonkgs/stonkgs and https://huggingface.co/stonkgs/stonkgs-150k .Keywords
All Related Versions
- Published version: Bioinformatics, 38 (6), 1648.
This publication has 18 references indexed in Scilit:
- A Survey on Knowledge Graphs: Representation, Acquisition, and ApplicationsIEEE Transactions on Neural Networks and Learning Systems, 2021
- Human pathways in animal models: possibilities and limitationsNucleic Acids Research, 2021
- Contextualized Protein-Protein InteractionsPatterns, 2021
- Enriching contextualized language model from knowledge graph for biomedical information extractionBriefings in Bioinformatics, 2020
- BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models2020
- From word models to executable models of signaling networks using automated assemblyMolecular Systems Biology, 2017
- Multimodal mechanistic signatures for neurodegenerative diseases (NeuroMMSig): a web server for mechanism enrichmentBioinformatics, 2017
- node2vec2016
- Automatic semantic classification of scientific literature according to the hallmarks of cancerBioinformatics, 2015
- DBpedia - A crystallization point for the Web of DataJournal of Web Semantics, 2009