Francesco Osborne

Ontologies and taxonomies of research fields are critical for managing and organising scientific knowledge, as they facilitate efficient classification, dissemination and retrieval of information. However, the creation and maintenance of such ontologies are expensive and time-consuming tasks, usually requiring the coordinated effort of multiple domain experts. Consequently, ontologies in this space often exhibit uneven coverage across different disciplines, limited inter-discipline connectivity, and infrequent updating cycles. In this study, we investigate the capability of several large language models to identify semantic relationships among research topics within three academic disciplines: biomedicine, physics, and engineering. The models were evaluated under three distinct conditions: zero-shot prompting, chain-of-thought prompting, and fine-tuning on existing ontologies. Additionally, we assessed the cross-discipline transferability of fine-tuned models by measuring their performance when trained in one discipline and subsequently applied to a different one. To support this analysis, we introduce PEM-Rel-8K , a novel dataset consisting of over 8,000 relationships extracted from the most widely adopted taxonomies in the three disciplines considered in this study: MeSH, PhySH, and IEEE. Our experiments demonstrate that fine-tuning LLMs on PEM-Rel-8K yields excellent performance across all disciplines.

Previous work has demonstrated that artificial intelligence (AI) methods for analysing scientific literature benefit significantly from annotating sentences in articles according to their rhetorical roles, such as research gaps, results, limitations, extensions of existing methodologies, and others. Such representations also have the potential to support the development of a new generation of systems capable of producing high-quality literature reviews. However, achieving this goal requires the definition of a relevant annotation schema and effective strategies for large-scale annotation of the literature. This article addresses these challenges in two ways: (1) it introduces a novel, unambiguous annotation schema that is explicitly designed for reliable automatic processing, and (2) it presents a comprehensive evaluation of a wide range of large language models (LLMs) on the task of classifying rhetorical roles according to this schema. To this end, we also present Sci-Sentence , a novel multidisciplinary benchmark comprising 700 sentences manually annotated by domain experts and 2,240 sentences automatically labelled using LLMs. We evaluate 37 LLMs on this benchmark, spanning diverse model families and sizes, using both zero-shot learning and fine-tuning approaches. The experiments reveal that modern LLMs achieve strong results on this task when fine-tuned on high-quality data, surpassing 96% F1, with both large proprietary models such as GPT-4o and lightweight open-source alternatives performing well. Moreover, augmenting the training set with semi-synthetic LLM-generated examples further boosts performance, enabling small encoders to achieve robust results and substantially improving several open decoder models.

In recent years, there has been a growing use of generative AI, and large language models (LLMs) in particular, to support both the assessment and generation of scientific work. Although some studies have shown that LLMs can, to a certain extent, evaluate research according to perceived quality, our understanding of the internal mechanisms that enable this capability remains limited. This paper presents the first study that investigates how LLMs encode the concept of scientific quality through relevant monosemantic features extracted using sparse autoencoders. We derive such features under different experimental settings and assess their ability to serve as predictors across three tasks related to research quality: predicting citation count, journal SJR, and journal h-index. The results indicate that LLMs encode features associated with multiple dimensions of scientific quality. In particular, we identify four recurring types of features that capture key aspects of how research quality is represented: 1) features reflecting research methodologies; 2) features related to publication type, with literature reviews typically exhibiting higher impact; 3) features associated with high-impact research fields and technologies; and 4) features corresponding to specific scientific jargons. These findings represent an important step toward understanding how LLMs encapsulate concepts related to research quality.

High-quality scientific research plays a pivotal role in advancing society, stimulating economic growth, protecting the environment, and driving technological innovation. Understanding the key factors that lead to impactful research is thus crucial as it can steer the development of more effective policies to enhance the research enterprise. Extensive literature emphasises that the composition of a research team is vital for generating innovative and impactful scientific work. Many studies have focused on how team diversity, including aspects like ethnicity, gender, and international background, affects research outcomes. These types of diversities often correlate positively with the impact of research. In this paper, we investigate a less-explored dimension of diversity: the diversity of authors’ areas of expertise. This aspect has received limited attention, primarily due to the challenges involved in defining and measuring it. We present new AI-driven methods to quantify this diversity of expertise and conduct an extensive analysis of over 944,000 Computer Science papers. Specifically, this study investigates the relationship between the authors’ diversity of expertise and the number of citations their paper receives within the first five years. For each paper, we modelled the expertise of each individual author and then quantified the overall diversity within the author team. We then performed a statistical analysis that revealed a significant positive correlation between two diversity metrics and the number of citations received. This suggests that, in the field of Computer Science, diversity of expertise is a key driver of high-impact research.

This study presents a method to analyze social media interactions and their relationship with investor decisions and stock market movements. It explores three questions: (1) the best method for classifying stock-related social media posts, (2) the influence of posts on stock returns and trading volumes, and (3) the impact of posts by opinion leaders. The WallstreetBets board on Reddit social media and the GameStop stock provide an ideal laboratory for answering these questions. We analyzed 135,000 posts, 5.6 million comments, and 90,000 users from November 2020 to June 2021. Using natural language processing and deep learning, posts and comments were classified by user intention (buy, sell, hold). We found that a BERT-based classifier achieved the highest performance with an F1 score of 80.2%. We then analyzed the relationship between social media activity and stock market data, finding that posts are related to trading volumes but not stock returns. However, considering the popularity of posts, there is a significant relationship between social media opinions and stock returns and trading volumes. The study highlights the importance of “influencers”: posts from a few influential users significantly impact stock returns and volumes. This research underscores the importance of social media dynamics in financial market behavior.