Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2312.09004

Holistic chemical evaluation reveals pitfalls in reaction prediction models

Victor Sabanza Gil·Andres M. Bran·Malte Franke·Remi Schlama·Jeremy S. Luterbacher·Philippe Schwaller
View original source ↗Read source document ↗Export metadata ↓

Abstract

The prediction of chemical reactions has gained significant interest within the machine learning community in recent years, owing to its complexity and crucial applications in chemistry. However, model evaluation for this task has been mostly limited to simple metrics like top-k accuracy, which obfuscates fine details of a model's limitations. Inspired by progress in other fields, we propose a new assessment scheme that builds on top of current approaches, steering towards a more holistic evaluation. We introduce the following key components for this goal: CHORISO, a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios, and a collection of metrics that provide a holistic view of a model's advantages and limitations. Application of this method to state-of-the-art models reveals important differences on sensitive fronts, especially stereoselectivity and chemical out-of-distribution generalization. Our work paves the way towards robust prediction models that can ultimately accelerate chemical discovery.

Explore related subjects

physics.chem-phcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Victor Sabanza Gil, Andres M. Bran, Malte Franke, Remi Schlama, Jeremy S. Luterbacher, Philippe Schwaller. 2023-12-14. Holistic chemical evaluation reveals pitfalls in reaction prediction models. https://arxiv.org/abs/2312.09004

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Global approximations to the error function of real argument for vectorized computation

The error function of real argument can be approximated to a given uniform relative accuracy by a single closed-form expression for the whole variable range either in terms of addition, multiplication, division, and square root operations only, or also using the exponential function. The coefficients have been tabulated for up to 128-bit precision. Tests of a computer code implementation using the standard single- and double-precision floating-point arithmetic show good performance and vectorizability. Approximations to the complementary error function have also been found, including those where only additions, multiplications, and one division are needed to reach a uniform absolute accuracy.

physics.chem-ph↗
arXiv2026

From Heuristics to Machine Learning: The Performance Ceiling for Single-Ion Magnets and Its Electronic Origin

Machine learning (ML) is expected to speed up the discovery of single-ion magnets (SIMs), but does the structural information available before synthesis allow such predictions? For 1215 lanthanide complexes from the SIMDAVIS 1.2.1 database we compared three increasing levels of structural description: tabular features of the coordination site, continuous symmetry measures of the coordination polyhedron, and the complete 3D arrangement of atoms. All three converge to an accuracy near 76%, only slightly above the 71% of the single rule "predict SIM for Dy3+". To explain the failures, we combined multireference ab initio calculations with an inspection of the structures behind the high-confidence errors. The SIMs missed by the geometric models are field-induced relaxers whose ground Kramers doublets are prone to tunnelling, a property invisible to geometric descriptors. Many false positives contain several lanthanide centers or radicals, so their relaxation is collective and outside the single-ion picture. The electronic-structure and connectivity information needed to identify SIMs is therefore not accessible to geometric methods alone. Geometric models remain useful: restricting the screening to compounds with high prediction confidence raises the accuracy to 88% while retaining 48% of the dataset. Building on the analysis of the failures, we propose a strategy that combines simple filters for nuclearity and for radicals with ligand-field descriptors from ab initio calculations.

physics.chem-ph↗
arXiv2026

Composition-Dependent Self-Diffusion Coefficients in Liquid Mixtures from Hybrid Machine Learning

Self-diffusion coefficients are key descriptors of molecular mobility, yet experimental data remain scarce, highlighting the need for reliable prediction methods. In previous work, we introduced the hybrid Enhanced Stokes-Einstein (ESE) model, which advanced the state of the art in the physically consistent prediction of self-diffusion coefficients of solutes at infinite dilution in pure solvents by integrating the Stokes-Einstein equation with machine learning (ML). Here, we extend this approach to concentration-dependent self-diffusion coefficients and multicomponent solvents with HADES. This hybrid architecture leverages a deep-set neural network to connect pure-component and mixture prediction within a single framework. HADES predicts self-diffusion coefficients in liquid mixtures with any number of components at any composition and temperature. The only required inputs are SMILES-encoded molecular structures of the components and the pure-component viscosities, making the method broadly applicable. Trained and evaluated on a comprehensive dataset of 2526 data points for 600 systems, HADES significantly outperforms benchmark prediction methods. The trained model and its source code are fully disclosed, and the application is available via an interactive website https://ml-prop.mv.rptu.de/.

physics.chem-ph↗