Serhii Starosyla

Public leaderboards such as the Therapeutics Data Commons (TDC) ADMET benchmark are widely treated as a ranking of state-of-the-art models. However, a high leaderboard position is only meaningful if the corresponding model can actually be reproduced and deployed by an independent researcher. In this work, we audit whether the top-ranked TDC ADMET models meet that bar. We assessed the top-ranked models of all 22 TDC ADMET leaderboards from the perspective of an end user with access only to the publicly released artifacts of each model─its publication, code repository, and installation instructions. For every end point, the top three models were screened with a unified protocol including an execution environment reproducibility check, a data-leakage assessment, verification of the hyperparameter-optimization procedure, and a reevaluation against the current leaderboard. Only three models (CaliciBoost, MapLight, and MapLight + GNN) passed all stages and reproduced their reported performance. The remaining models failed because of unavailable code, nonreproducible environments, runtime incompatibilities, or methodological flaws. We traced direct or indirect data leakage in the MiniMol, GradientBoost, and XGBoost models, and used deliberately overfitted variants of our own Mol2Vec-based models to show that tuning on the public test set─whether accidental or intentional─can substantially inflate both metrics and leaderboard rank. These results indicate that current TDC leaderboard positions cannot be read as a direct measure of model quality and practical applicability and emphasize the urgent need for better public ADMET benchmarks based on the hidden test sets, strict data set versioning and model submission with standardized inference environments.

In this work we performed a critical assessment of the benchmarking procedures used in Therapeutics Data Commons (TDC) ADMET leaderboards, focusing on reproducibility, robustness against data leakage, and signs of test-set overfitting across all 22 TDC ADMET endpoints. For each endpoint, the top 3 leaderboard models were screened with a unified protocol: execution environment reproducibility check, data leakage assessment, verification of hyperparameter optimisation practices, and final re-evaluation of results and TDC ranking. Only 3 methods (CaliciBoost, MapLight, MapLight+GNN) passed all checks and showed overall reproducible performance, whereas most of top-ranked models exhibited unavailable code, non-reproducible execution environments, runtime incompatibilities, or various methodological flaws. In particular, we identified direct or indirect data leakages in MiniMol, GradientBoost and XGBoost models. We also used our in-house models based on the Mol2Vec architecture to investigate the consequences of deliberately overfitting on the TDC test set. It is shown that deliberate or accidental tuning on the public test set may lead to significant inflation of the model metrics and leaderboard position. Our results emphasize the urgent need for better public ADMET benchmarks with the hidden test sets, strict dataset versioning and model submission with standardized inference environments.

Classical protein-ligand docking has been a cornerstone technique in computational drug discovery for decades but has reached an accuracy and performance plateau. Recently introduced Machine Learning (ML)-based docking methods offer a promising paradigm shift, but their practical adoption is hampered by accuracy-to-speed trade-offs, inadequate benchmarking standards, and questionable chemical validity of predicted poses. In this study, we introduce ArtiDock─an ML-based docking technique optimized for high-throughput virtual screening applications. To evaluate ArtiDock, we developed a dedicated performance and accuracy benchmark for pocket-specific rigid protein-ligand docking, which mimics realistic industrial drug discovery scenarios and is based on the novel PLINDER data set. We demonstrate that ArtiDock is 29-38% more accurate in comparison to leading open-source and commercial classical docking techniques such as AutoDock, Vina, and Glide, while providing a low computational cost. ArtiDock notably excels in challenging docking scenarios involving unbound protein structures and binding sites containing ions and structured water molecules. Additionally, we demonstrated competitive accuracy of our approach at considerably higher throughput compared to a wide range of AI docking and AI cofolding methods using the PoseX benchmark. Our results show that ArtiDock could be considered as a method of choice in high-throughput virtual screening scenarios.

Rigid docking remains the most popular method of predicting protein-protein interactions in cases when experimental 3D structures of the complexes are not available. The docking often relies on known unbound (Apo) protein structures, which may differ significantly from their bound (Holo) forms. Modern machine learning (ML) based conformational sampling techniques allow generating ensembles of functionally relevant protein structures, which may be closer to their Holo forms and thus could improve the outcomes of the classical rigid protein-protein docking. Here, we sampled conformations of the protein subunits in 30 complexes from the novel PINDER data set with two state-of-the-art ML-based techniques and evaluated their docking performance using several physics-based, data-based, and ML-based scoring functions. We showed that such conformational sampling rarely produces structures that are closer to the Holo conformations than the corresponding Apo ones. Moreover, even when such conformations are generated, none of the tested scoring functions were able to prioritize and rank them correctly. Our work highlights critical limitations in the current ML-enhanced rigid protein-protein docking workflows and emphasizes the need for new approaches that can better utilize the potential of modern techniques for conformational generation and scoring.

Motivation: Accurately identifying and prioritizing protein binding pockets is a foundational element of small-molecule drug discovery. Defining these known pockets currently relies on a laborious manual process of extracting key residue data from selected publications, reconciling inconsistent terminology, and independently computing volumetric representations. This manual curation to ensure biological relevance is time-consuming, error-prone, and represents a major bottleneck for efficient, high-throughput drug discovery. Results: We present a novel approach for the identification and prioritization of protein binding pockets for small molecules by combining geometric pocket detection with large language models (LLMs). Our method leverages Fpocket to generate candidate pockets, which are then validated against published experimental data extracted from research articles using LLM with a series of prompts fine-tuned to identify and extract residue-level information associated with experimentally confirmed binding sites. We developed a curated benchmark dataset of diverse proteins and associated literature to train and evaluate the LLM’s performance in paper relevance assessment and pocket extraction. Availability and implementation: The developed benchmark dataset and methodology are freely available at the GitHub repository (https://github.com/receptor-ai/LLM-benchmark-dataset) and Zenodo (DOI: 10.5281/zenodo.15798647).

The accurate prediction of ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) properties is crucial for early-stage drug development, enabling the reduction of late-stage attrition and guiding compound prioritization. In recent years, machine learning models have emerged as powerful tools for ADMET prediction, leveraging diverse molecular representations ranging from handcrafted descriptors to graph neural networks and language model embeddings. Despite these advances, balancing predictive performance with computational efficiency remains a key challenge, particularly for high-throughput screening scenarios. Among unsupervised embedding methods, Mol2Vec has shown promise by capturing chemical substructure context analogously to word embeddings in natural language processing. However, its performance on comprehensive ADMET benchmarks has not been systematically assessed. In this work, we reimplement Mol2Vec with an expanded training corpus and higher embedding dimensionality, and evaluate its utility across 16 ADMET prediction tasks from the Therapeutics Data Commons (TDC). We show that while Mol2Vec embeddings alone are competitive, combining them with classical molecular descriptors and applying feature selection significantly improves performance. Our final MLP models with enhanced Mol2Vec embeddings achieved top-1 results in 10 of 16 benchmarks, outperforming all previously reported models on the TDC leaderboard in this regard, demonstrating that descriptor-enriched representations, paired even with relatively simple MLPs, can rival or exceed the performance of more complex models.

Rigid docking remains the most popular method of predicting protein-protein interactions in cases when experimental 3D structures of the complexes are not available. The docking often relies on known unbound (Apo) protein structures, which may differ significantly from their bound (Holo) forms. Modern machine learning (ML) based conformational sampling techniques allow generating ensembles of functionally relevant protein structures, which may be closer to their Holo forms and thus could improve the outcomes of the classical rigid protein-protein docking. Here, we sampled conformations of the protein subunits in 30 complexes from the novel PINDER dataset with two state-of-the-art ML-based techniques and evaluated their docking performance using several physics-based, data-based, and ML-based scoring functions. We showed that such conformational sampling rarely produces structures that are closer to the Holo conformations than the corresponding Apo ones. Moreover, even when such conformations are generated, none of the tested scoring functions were able to prioritize and rank them correctly. Our work highlights critical limitations in the current ML-enhanced rigid protein-protein docking workflows and emphasizes the need for new approaches that can better utilize the potential of modern techniques for conformational generation and scoring.

We present a novel approach for the identification and prioritization of protein binding pockets for small molecules by combining geometric pocket detection with Large Language Models (LLMs). Our method leverages Fpocket to generate candidate pockets, which are then validated against published experimental data extracted from research articles using LLM with a series of prompts fine-tuned to identify and extract residue-level information associated with experimentally confirmed binding sites. We developed a curated benchmark dataset of diverse proteins and associated literature to train and evaluate the LLM’s performance in paper relevance assessment and pocket extraction. The extracted information is then mapped onto protein structures and used to filter and merge the geometry-based predictions, generating a refined volumetric representation of biologically relevant pockets. This hybrid pipeline offers an efficient, accurate and automated method for identifying functional binding pockets, addressing a significant bottleneck in the high-throughput drug discovery workflows. The developed benchmark dataset and methodology are freely available at https://github.com/MelnychenkoM/LLM-benchmark-dataset .

Classical protein-ligand docking has been a cornerstone technique in computational drug discovery for decades, but has reached an accuracy and performance plateau. Recently introduced Machine Learning (ML) based docking methods offer a promising paradigm shift, but their practical adoption is hampered by accuracy-to-speed trade-offs, inadequate benchmarking standards, and questionable chemical validity of predicted poses. In this study, we introduce ArtiDock - an ML-based docking technique optimized for high-throughput virtual screening applications. To evaluate ArtiDock, we developed a dedicated performance and accuracy benchmark for pocket-specific rigid protein-ligand docking, which mimics realistic industrial drug discovery scenarios and is based on the novel PLINDER dataset. We demonstrate that ArtiDock is 29-38% more accurate in comparison to leading open-source and commercial classical docking techniques such as AutoDock, Vina, and Glide, while providing a low computational cost. ArtiDock notably excels in challenging docking scenarios involving unbound protein structures and binding sites containing ions and structured water molecules. Our results show that ArtiDock could be considered as a method of choice in high-throughput virtual screening scenarios.

This study introduces the PocketCFDM generative diffusion model, aimed at improving the prediction of small molecule poses in the protein binding pockets. The model utilizes a novel data augmentation technique, involving the creation of numerous artificial binding pockets that mimic the statistical patterns of non-bond interactions found in actual protein–ligand complexes. An algorithmic method was developed to assess and replicate these interaction patterns in the artificial binding pockets built around small molecule conformers. It is shown that the integration of artificial binding pockets into the training process significantly enhanced the model's performance. Notably, PocketCFDM surpassed DiffDock in terms of non-bond interaction and steric clash numbers, and the inference speed. Future developments and optimizations of the model are discussed. The inference code and final model weights of PocketCFDM are accessible publicly via the GitHub repository: https://github.com/vtarasv/pocket-cfdm.git.