Africanus IV. The Stimela2 framework: scalable and reproducible workflows, from local to cloud compute

Oleg M. Smirnov; Sphesihle Makhathini; Jonathan S. Kenyon; Hertzog L. Bester; Simon J. Perkins; Athanaseus J. T. Ramaila; Benjamin V. Hugo

arXiv · 2412.10080

Africanus IV. The Stimela2 framework: scalable and reproducible workflows, from local to cloud compute

Oleg M. Smirnov·Sphesihle Makhathini·Jonathan S. Kenyon·Hertzog L. Bester·Simon J. Perkins·Athanaseus J. T. Ramaila·Benjamin V. Hugo

View original source ↗Read source document ↗Export metadata ↓

Abstract

Stimela2 is a new-generation framework for developing data reduction workflows. It is designed for radio astronomy data but can be adapted for other data processing applications. Stimela2 aims at the middle ground between ease of development, human readability, and enabling robust, scalable and reproducible workflows. It represents workflows by linear, concise and intuitive YAML-format "recipes". Atomic data reduction tasks (binary executables, Python functions and code, and CASA tasks) are described by YAML-format "cab definitions" detailing each task's "schema" (inputs and outputs). Stimela2 provides a rich syntax for chaining tasks together, and encourages a high degree of modularity: recipes may be nested into other recipes, and configuration is cleanly separated from recipe logic. Tasks can be executed natively or in isolated environments using containerization technologies such as Apptainer. The container images are open-source and maintained through a companion package called cult-cargo. This enables the development of system-agnostic and fully reproducible workflows. Stimela2 facilitates the deployment of scalable, distributed workflows by interfacing with the Slurm scheduler and the Kubernetes API. The latter allows workflows to be readily deployed in the cloud. Previous papers in this series used Stimela2 as the underlying technology to run workflows on the AWS cloud. This paper presents an overview of Stimela2's design, architecture and use in the radio astronomy context.

Explore related subjects

astro-ph.IM

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Oleg M. Smirnov, Sphesihle Makhathini, Jonathan S. Kenyon, Hertzog L. Bester, Simon J. Perkins, Athanaseus J. T. Ramaila, Benjamin V. Hugo. 2024-12-17. Africanus IV. The Stimela2 framework: scalable and reproducible workflows, from local to cloud compute. https://arxiv.org/abs/2412.10080

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Radial Pulsations in Polaris: A Secondary Science Application of Cherenkov Telescopes via Intensity Interferometry

Ground-based Cherenkov telescopes, which are typically inoperative during moonlit nights for gamma-ray observations, offer a valuable opportunity during this time for secondary scientific applications through Intensity Interferometry (II). Recent developments and observations suggest that implementing II instrumentation on existing and planned Imaging Atmospheric Cherenkov Telescopes (IACTs) can significantly advance optical stellar measurements. Motivated by the resurgence of II efforts over the past two decades, this work presents simulations demonstrating the estimation of stellar parameters for a radially pulsating star, such as Polaris, using either a single telescope or multiple telescopes. For single-telescope simulations, we assume that the photon pixels in the camera are mapped onto four distinct regions of the aperture, generating multiple baselines and enabling enhanced observational plane coverage. These results highlight the potential of Cherenkov telescopes in India for high-resolution optical astronomy during otherwise inoperative periods and offer promising insights into the characterization of bright stellar objects with unprecedented precision.

astro-ph.IM↗

arXiv2026

Parallel adaptive reweighting importance sampling for Bayesian astrophysics

Efficient sampling from high-dimensional, multi-modal posteriors is a central challenge in Bayesian inference across the physical sciences. Gravitational-wave (GW) astronomy, where likelihood evaluations are computationally expensive and posteriors are complex, is one application motivating this work. Popular families of methods like Markov-chain Monte Carlo, nested sampling, and importance sampling all rely on proposal distributions to guide exploration. Because prior knowledge of the target is often limited, practitioners can adopt adaptive proposals that iteratively refine themselves using information gained from previously drawn samples. Traditional adaptive strategies, however, struggle in high-dimensional multi-modal settings: complex, non-linear correlations are hard to capture, and hyperparameters typically require tedious, problem-specific tuning. To address these issues, we introduce Parallel Adaptive Reweighting Importance Sampling (PARIS; descriptively, ``seed sampling''). PARIS models its proposal as a Gaussian mixture whose component centers are the existing samples and whose component weights match the current importance weights. New draws from the proposal therefore concentrate around high-weight regions, while candidate points in unexplored areas receive intentionally inflated weights. As the algorithm continuously reweights all samples up to the latest proposal, any initial over-weighting self-corrects once additional neighbor samples are collected. To enable rapid reweighting, we present an efficient update scheme and evaluate PARIS on illustrative toy problems and more realistic gravitational-wave parameter estimation tasks. PARIS achieves accurate posterior reconstruction and evidence estimation with substantially fewer function evaluations than competing approaches, highlighting its promise for widespread use in astrophysical data analysis.

astro-ph.IM↗

arXiv2026

Ising noise filter: physics-informed filtering for particle detectors

We present the Ising noise filter, a highly portable, graph-based pre-filtering algorithm for early-stage background suppression in particle accelerators and astrophysical detectors. Standard noise rejection methods relying on track fitting suffer from severe combinatorial explosion. Our method bypasses this by mapping individual detector hits to a network of binary spins and minimizing an energy functional. The interaction kernels are physics-informed, tailored to the underlying physics and geometry of the experiment. We demonstrate the efficacy of this approach in two distinct experimental regimes. Applied to the Baikal-GVD neutrino telescope the filter yields fast, standard-quality noise rejection with 96.8% recall for astrophysical neutrinos. For the SPD detector at the NICA collider the filter attains recall of 97% on a toy Monte Carlo sample. Furthermore, when combined with a Peterson--Hopfield network for track finding, our physics-informed coupling improves the TrackML score from 0.5 to 0.95.

astro-ph.IM↗