Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 1505.05617

Data Compression in the Petascale Astronomy Era: a GERLUMPH case study

Dany Vohl·Christopher J. Fluke·Georgios Vernardos
View original source ↗Read source document ↗Export metadata ↓

Abstract

As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -- and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ~27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -- well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.

Explore related subjects

astro-ph.IM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Dany Vohl, Christopher J. Fluke, Georgios Vernardos. 2016-05-05. Data Compression in the Petascale Astronomy Era: a GERLUMPH case study. https://doi.org/10.1016/j.ascom.2015.05.003

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Performance characterization of a new Structural and Thermal Architecture for a future spaceborne Closed-Cycle Dilution Refrigerator

A Structural and Thermal Model (STM) has been developed to support the new spaceborne Closed-Cycle Dilution Refrigerator (CCDR), which aims to provide continuous cooling at 100~mK for long-duration astrophysical missions. The STM is based on a hexapod architecture that ensures both thermal decoupling and mechanical robustness during launch. In this paper, we present the characterization of its thermal and mechanical performances. A dedicated experimental setup was used to investigate the thermal behavior of the STM across a broad temperature range. The study reveals limitations of the collar design, with incomplete power interception from thermal boundary resistances and vibration test failure traced to defective strut gluing. These results guide the next STM iteration with optimized collar and strut assembly for reliable CCDR operation in space.

astro-ph.IM↗
arXiv2026

The Simons Observatory: Development of a Pipeline to Detect Rapid Transients in Time-Ordered Data

We introduce a method for detecting astrophysical transients evolving on timescales of milliseconds to minutes using cosmic microwave background (CMB) survey telescopes. While previous transient searches in CMB data operate in map space, our pipeline directly processes the raw time-ordered data, enabling sensitivity to fast, dynamic signals. We integrate our detection approach into the Simons Observatory time-domain pipeline and assess the performance by injecting symmetric, stellar flare-like light curves into simulated observations. For events flaring with a timescale of 0.5 s, the pipeline detects $\gtrsim90$ % of events at flux densities of 800, 1150, 1650, and 4250\,mJy when measured in the 93, 145, 225, and 280 GHz bands respectively. At a fixed peak flux density, the pipeline more readily detects longer flares. The limiting flux density for 90 % completeness is four times lower for a $\ge5$ s flare than for a 0.5 s flare, while the flux density limits for $\gtrsim50$ % detection efficiency are comparable to the rms noise of the time-ordered data. We are able to determine the position of detected events in each observing band, with a positional uncertainty at the detection threshold comparable to the telescope resolution at that band. These results demonstrate the readiness of this pipeline for incorporation into upcoming Simons Observatory data analyses.

astro-ph.IM↗
arXiv2026

Thermal conductivity of various CFRPs from 100 mK to 20 K

Carbon-fiber-reinforced polymers (CFRPs) are some of the most useful materials for building spacecraft and aerospace tools. They are especially valuable for systems that work at extremely cold (cryogenic) temperatures because they are strong, lightweight, and don't transfer heat easily. In this study, researchers measured how well heat moves through several different types of carbon fiber samples, specifically T300, T700, HS40, M55J, and IMA, at different fiber layouts and densities. These measurements were taken at ultra-cold temperatures ranging from 100 mK to 20 K. The team used a newly developed analysis method to calculate the thermal conductivity for each sample. Finally, they shared how each material behaved at different temperatures and compared their findings to previous research.

astro-ph.IM↗