Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2311.16321

A Ceph S3 Object Data Store for HEP

Nick Smith·Bo Jayatilaka·David Mason·Oliver Gutsche·Alison Peisker·Robert Illingworth·Chris Jones
View original source ↗Read source document ↗Export metadata ↓

Abstract

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

Explore related subjects

physics.data-anhep-ex

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Nick Smith, Bo Jayatilaka, David Mason, Oliver Gutsche, Alison Peisker, Robert Illingworth, Chris Jones. 2023-11-27. A Ceph S3 Object Data Store for HEP. https://arxiv.org/abs/2311.16321

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Simulation-Based Inference and Unbinned Asimov Construction with Hybrid Neural Density Estimation

High-dimensional, unbinned neural simulation-based inference often relies on neural ratio estimation, which uses expressive supervised models to estimate density ratios, but a learned ratio by itself provides neither an explicit normalized density nor a generative model. Flow-based surrogate models instead enable tractable density evaluation and efficient sampling, but residual density-estimation errors can limit precision for complex implicit distributions. We propose \textit{hybrid neural density estimation}, which uses a flow to define a parameter-independent reference distribution, and classifiers to estimate target-to-reference density ratios. Multiplying a learned ratio by the reference density gives an evaluable target density surrogate. The ratio also provides importance weights for integration and resampling. We show how this representation defines an exact Asimov dataset, whose maximum likelihood fit returns the generating parameters. We also show how the tractable reference supplies renewable samples for pseudo-experiments and how it enables methods to reduce the Monte Carlo variance of expected test statistic calculations. We demonstrate the construction in a toy statistical model motivated by high-energy physics measurements, but for which the exact densities are known analytically.

physics.data-an↗
arXiv2026

Learning to bin: differentiable and Bayesian optimization for multi-dimensional discriminants in high-energy physics

Categorizing events using discriminant observables is central to many high-energy physics analyses. Yet, bin boundaries are often chosen manually. A simple, popular choice in multi-classification tasks is to assign events according to the largest per-class score ("argmax") and to apply equidistant binning to the resulting one-dimensional discriminants. We propose a binning optimization for signal significance directly in multi-dimensional discriminants. We use a Gaussian Mixture Model (GMM) to define flexible regions in the score space, which can be interpreted either as bins or as analysis categories. While this GMM-based strategy is applicable in both one and multiple dimensions, we also study a direct bin-boundary optimization in one dimension as a simpler alternative for binary discriminants. On this binning model, we study two optimization strategies: a differentiable and a Bayesian optimization approach. We study two toy setups: a binary classification and a three-class problem with two signals and backgrounds. In the one-dimensional case, both approaches achieve similar gains in signal sensitivity compared to equidistant binning for a given number of bins, while in the multi-dimensional case the differentiable approach performs best. We show that the GMM-based optimization can outperform argmax classification even after optimized binning is applied to the one-dimensional projections. We further study the performance of our methods on the FAIR Universe $H\rightarrowττ$ dataset, where the GMM-based optimization gives the highest signal significance. Both methods are released as lightweight Python plugins intended for straightforward integration into existing analyses.

physics.data-an↗
arXiv2026

When Should Team KPIs Be Absolute or Relative for Match-Outcome Prediction?

In rugby union and association football, team key performance indicators (KPIs) can be represented in absolute terms or relative to the opponent. Relativisation sometimes improves match-outcome prediction and sometimes harms it. There has been no general account of when each occurs. The answer depends on how much the two teams differ in variability and how strongly their KPI values rise and fall together. We combine these properties into the Paired Efficiency Factor (PEF), which generalises Fisher's paired-efficiency result to the unequal-variance conditions typical of competitive sport. The PEF also connects a KPI's statistical efficiency to how much information its relative form carries about the outcome. Combined KPIs can interact in complex ways, so we analyse each indicator on its own. Across 86 team KPIs from professional rugby union and association football, the PEF places every metric in one of four regimes. Anti-correlation is common, especially for high-volume competitive counts, and absolute measures are then usually preferable. Relativisation can still improve prediction when a noisier difference carries more outcome information. An idealised simulation and one representative KPI from each regime confirm both signs under team-blocked cross-validation. The same pairing geometry appears in healthcare, genomics, finance, and manufacturing. The PEF turns an ad hoc feature-engineering choice into a transparent, data-informed diagnostic for when to relativise performance metrics and when not to.

physics.data-an↗