Daniel Ernesto Acuna

Open scholarly metadatabases are becoming an indispensable component of quantitative studies of science. In particular, reference information from these databases is essential for citation analysis, including impact evaluation and disruption measurement. Recent studies have examined the reference coverage of different scholarly databases; nevertheless, the underlying factors driving the disparity in reference quality across publications remain largely unexplored. Here, we examine the reference quality of OpenAlex, one of the largest open scholarly metadatabases to date, and find pervasive reference missingness. Importantly, we observe that this effect is more pronounced in articles associated with less prestigious institutions, lower impact journals, and authors affiliated with countries of lower economic standing. A striking 57% of articles below the median prestige and journal impact that belong to the Global South are completely missing references. A regression analysis revealed that these disparities are mainly associated with the prestige of the journal and the institution. We discuss how we should address this issue more systematically and exercise caution when using metadata to assess impact.

Questionable journals threaten global research integrity, yet manual vetting can be slow and inflexible. Here, we explore the potential of artificial intelligence (AI) to systematically identify such venues by analyzing website design, content, and publication metadata. Evaluated against extensive human-annotated datasets, our method achieves practical accuracy and uncovers previously overlooked indicators of journal legitimacy. By adjusting the decision threshold, our method can prioritize either comprehensive screening or precise, low-noise identification. At a balanced threshold, we flag over 1000 suspect journals, which collectively publish hundreds of thousands of articles, receive millions of citations, acknowledge funding from major agencies, and attract authors from developing countries. Error analysis reveals challenges involving discontinued titles, book series misclassified as journals, and small society outlets with limited online presence, which are issues addressable with improved data quality. Our findings demonstrate AI’s potential for scalable integrity checks, while also highlighting the need to pair automated triage with expert review.

The rapid advancement of Natural Language Processing (NLP) has greatly improved text-generation tools like ChatGPT and Claude, offering significant utility but also posing risks to media credibility through paraphrased plagiarism—a subtle yet widespread form of content misuse. Despite progress in automated paraphrase detection, inconsistencies in training datasets often limit their effectiveness. This study examines traditional and modern approaches to paraphrase identification, revealing how the under-representation of certain paraphrase types in widely-used datasets, including those for training Large Language Models (LLMs), undermines plagiarism detection accuracy. To address these issues, we introduce and validate ReParaphrased, a refined paraphrase typology, and extend the Extended Typology Paraphrase Corpus (ETPC) with meticulous manual annotations to enhance reliability. Using the augmented ETPC, we fine-tune the LLama3.1-7B-instruct model, uncovering significant disparities in paraphrase type distribution across existing datasets. A detailed analysis of the MRPC benchmark dataset further highlights critical distributional issues and their implications. We propose four key solutions to address dataset limitations, providing both theoretical and practical guidance for improving dataset quality. These contributions aim to establish a more robust foundation for NLP model training and evaluation. Finally, we outline future research directions and suggest improvements for dataset development to advance AI-driven paraphrase detection.

Correctness is a key aspiration of the scientific process, yet recent studies suggest that many high-profile findings may be difficult to replicate or require considerable evidence for verification. Proposals to fix these issues typically ask for tighter statistical controls (e.g., stricter p-value thresholds or higher statistical power). However, these approaches often overlook the importance of contemplating research outcomes’ potential costs and benefits. Here, we develop a framework grounded in Bayesian decision theory that seamlessly integrates cost-benefit analysis into evaluating research programs with potentially uncertain results. We derive minimally acceptable prestudy odds and positive predictive values for cost and benefit levels. We show that tolerance to inaccurate results changes dramatically due to uncertainties posed by research. We also show that reducing uncertainties (e.g., by recruiting more subjects) may have limited effects on the expected benefit of continuing specific research programs. We apply our framework to several types of cancer research and their funding. Our analysis shows that highly exploratory research designs are easily justifiable due to their potential benefits, even when probabilistic models suggest otherwise. We discuss how the cost and benefit of research could and should always be part of the toolkit used by scientists, institutions, or funding agencies.

Datasets are critical for scientific research, playing an important role in replication, reproducibility, and efficiency. Researchers have recently shown that datasets are becoming more important for science to function properly, even serving as artifacts of study themselves. However, citing datasets is not a common or standard practice in spite of recent efforts by data repositories and funding agencies. This greatly affects our ability to track their usage and importance. A potential solution to this problem is to automatically extract dataset mentions from scientific articles. In this work, we propose to achieve such extraction by using a neural network based on a Bi-LSTM-CRF architecture. Our method achieves F1 = 0.885 in social science articles released as part of the Rich Context Dataset. We discuss the limitations of the current datasets and propose modifications to the model to be done in the future.

Obtaining funding is an important part of becoming a successful scientist. Junior faculty spend a great deal of time finding the right agencies and programs that best match their research profile. But what are the factors that influence the best publication--grant matching? Some universities might employ pre-award personnel to understand these factors, but not all institutions can afford to hire them. Historical records of publications funded by grants can help us understand the matching process and also help us develop recommendation systems to automate it. In this work, we present \textsc{GotFunding} (Grant recOmmendaTion based on past FUNDING), a recommendation system trained on National Institutes of Health's (NIH) grant--publication records. Our system achieves a high performance (NDCG@1 = 0.945) by casting the problem as learning to rank. By analyzing the features that make predictions effective, our results show that the ranking considers most important 1) the year difference between publication and grant grant, 2) the amount of information provided in the publication, and 3) the relevance of the publication to the grant. We discuss future improvements of the system and an online tool for scientists to try.

Scientist learn early on how to cite scientific sources to support their claims. Sometimes, however, scientists have challenges determining where a citation should be situated -- or, even worse, fail to cite a source altogether. Automatically detecting sentences that need a citation (i.e., citation worthiness) could solve both of these issues, leading to more robust and well-constructed scientific arguments. Previous researchers have applied machine learning to this task but have used small datasets and models that do not take advantage of recent algorithmic developments such as attention mechanisms in deep learning. We hypothesize that we can develop significantly accurate deep learning architectures that learn from large supervised datasets constructed from open access publications. In this work, we propose a Bidirectional Long Short-Term Memory (BiLSTM) network with attention mechanism and contextual information to detect sentences that need citations. We also produce a new, large dataset (PMOA-CITE) based on PubMed Open Access Subset, which is orders of magnitude larger than previous datasets. Our experiments show that our architecture achieves state of the art performance on the standard ACL-ARC dataset ($F_{1}=0.507$) and exhibits high performance ($F_{1}=0.856$) on the new PMOA-CITE. Moreover, we show that it can transfer learning across these datasets. We further use interpretable models to illuminate how specific language is used to promote and inhibit citations. We discover that sections and surrounding sentences are crucial for our improved predictions. We further examined purported mispredictions of the model, and uncovered systematic human mistakes in citation behavior and source data. This opens the door for our model to check documents during pre-submission and pre-archival procedures. We make this new dataset, the code, and a web-based tool available to the community.

Pawin Taechoyotin, Daniel Acuna. Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024). 2024.

Alexandria Leto, Shamik Roy, Alexander Hoyle, Daniel Acuna, Maria Leonor Pacheco. Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS 2024). 2024.