Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 1807.05733

Pipeline Collector: gathering performance data for distributed astronomical pipelines

Alexandar P. Mechev·Aske Plaat·J. B. Raymond Oonk·Huib T. Intema·Huub J. A. Röttgering
View original source ↗Read source document ↗Export metadata ↓

Abstract

Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.

Explore related subjects

astro-ph.IM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Alexandar P. Mechev, Aske Plaat, J. B. Raymond Oonk, Huib T. Intema, Huub J. A. Röttgering. 2018-07-16. Pipeline Collector: gathering performance data for distributed astronomical pipelines. https://doi.org/10.1016/j.ascom.2018.06.005

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Performance characterization of a new Structural and Thermal Architecture for a future spaceborne Closed-Cycle Dilution Refrigerator

A Structural and Thermal Model (STM) has been developed to support the new spaceborne Closed-Cycle Dilution Refrigerator (CCDR), which aims to provide continuous cooling at 100~mK for long-duration astrophysical missions. The STM is based on a hexapod architecture that ensures both thermal decoupling and mechanical robustness during launch. In this paper, we present the characterization of its thermal and mechanical performances. A dedicated experimental setup was used to investigate the thermal behavior of the STM across a broad temperature range. The study reveals limitations of the collar design, with incomplete power interception from thermal boundary resistances and vibration test failure traced to defective strut gluing. These results guide the next STM iteration with optimized collar and strut assembly for reliable CCDR operation in space.

astro-ph.IM↗
arXiv2026

The Simons Observatory: Development of a Pipeline to Detect Rapid Transients in Time-Ordered Data

We introduce a method for detecting astrophysical transients evolving on timescales of milliseconds to minutes using cosmic microwave background (CMB) survey telescopes. While previous transient searches in CMB data operate in map space, our pipeline directly processes the raw time-ordered data, enabling sensitivity to fast, dynamic signals. We integrate our detection approach into the Simons Observatory time-domain pipeline and assess the performance by injecting symmetric, stellar flare-like light curves into simulated observations. For events flaring with a timescale of 0.5 s, the pipeline detects $\gtrsim90$ % of events at flux densities of 800, 1150, 1650, and 4250\,mJy when measured in the 93, 145, 225, and 280 GHz bands respectively. At a fixed peak flux density, the pipeline more readily detects longer flares. The limiting flux density for 90 % completeness is four times lower for a $\ge5$ s flare than for a 0.5 s flare, while the flux density limits for $\gtrsim50$ % detection efficiency are comparable to the rms noise of the time-ordered data. We are able to determine the position of detected events in each observing band, with a positional uncertainty at the detection threshold comparable to the telescope resolution at that band. These results demonstrate the readiness of this pipeline for incorporation into upcoming Simons Observatory data analyses.

astro-ph.IM↗
arXiv2026

Thermal conductivity of various CFRPs from 100 mK to 20 K

Carbon-fiber-reinforced polymers (CFRPs) are some of the most useful materials for building spacecraft and aerospace tools. They are especially valuable for systems that work at extremely cold (cryogenic) temperatures because they are strong, lightweight, and don't transfer heat easily. In this study, researchers measured how well heat moves through several different types of carbon fiber samples, specifically T300, T700, HS40, M55J, and IMA, at different fiber layouts and densities. These measurements were taken at ultra-cold temperatures ranging from 100 mK to 20 K. The team used a newly developed analysis method to calculate the thermal conductivity for each sample. Finally, they shared how each material behaved at different temperatures and compared their findings to previous research.

astro-ph.IM↗