LitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs

Authors:
  • JZJiasheng Zhang
    Jiasheng Zhang
  • JCJialin Chen
    Jialin Chen
  • AMAli Maatouk
    Ali Maatouk
  • NBNgoc Bui
    Ngoc Bui
  • QXQianqian Xie
    Qianqian Xie
  • LTLeandros Tassiulas
    Leandros Tassiulas
  • JSJie Shao
    Jie Shao
  • HXHua Xu
    Hua Xu
  • RYRex Ying
    Rex Ying

Abstract
With the advent of large language models (LLMs), managing scientific literature via LLMs has become a promising direction of research. However, existing approaches often overlook the rich structural and semantic relevance among scientific literature, limiting their ability to discern the relationships between pieces of scientific knowledge, and suffer from various types of hallucinations. These methods also focus narrowly on individual downstream tasks, limiting their applicability across use cases. Here we propose LitFM, the first literature foundation model designed for a wide variety of practical downstream tasks on domain-specific literature, with a focus on citation information. At its core, LitFM contains a novel graph retriever to integrate graph structure by navigating citation graphs and extracting relevant literature, thereby enhancing model reliability. LitFM also leverages a knowledge-infused LLM, fine-tuned through a well-developed instruction paradigm. It enables LitFM to extract domain-specific knowledge from literature and reason relationships among them. By integrating citation graphs during both training and inference, LitFM can generalize to unseen papers and accurately assess their relevance within existing literature. Additionally, we introduce new large-scale literature citation benchmark datasets on three academic fields, featuring sentence-level citation information and local context. Extensive experiments validate the superiority of LitFM, achieving 28.1% improvement on retrieval task in precision, and an average improvement of 7.52% over state-of-the-art across six downstream literature-related tasks
All Related Versions