Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 1611.06039

Pre-feasibility Study of Astronomical Data Archive Systems Powered by Public Cloud Computing and Hadoop Hive

Satoshi Eguchi
View original source ↗Read source document ↗Export metadata ↓

Abstract

The size of astronomical observational data is increasing yearly. For example, while Atacama Large Millimeter/submillimeter Array is expected to generate 200 TB raw data every year, Large Synoptic Survey Telescope is estimated to produce 15 TB raw data every night. Since the increasing rate of computing is much lower than that of astronomical data, to provide high performance computing (HPC) resources together with scientific data will be common in the next decade. However, the installation and maintenance costs of a HPC system can be burdensome for the provider. I note public cloud computing for an alternative way to get sufficient computing resources inexpensively. I build Hadoop and Hive clusters by utilizing a virtual private server (VPS) service and Amazon Elastic MapReduce (EMR), and measure their performances. The VPS cluster behaves differently day by day, while the EMR clusters are relatively stable. Since partitioning is essential for Hive, several partitioning algorithms are evaluated. In this paper, I report the results of the benchmarks and the performance optimizations in cloud computing environment.

Explore related subjects

astro-ph.IM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Satoshi Eguchi. 2016-11-18. Pre-feasibility Study of Astronomical Data Archive Systems Powered by Public Cloud Computing and Hadoop Hive. https://arxiv.org/abs/1611.06039

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Scale-Vector Alignment: A Scale-Aware Framework for Spatially Resolved Morphological Similarity in Astronomical Images

Astronomical maps made with different tracers are not expected to have identical morphology. Excitation, optical depth, chemistry, radiation, and ISM phase alter the response of a tracer, and the resulting differences can depend on both position and spatial scale. We propose scale-vector alignment, a scale-aware method based on Constrained Diffusion Decomposition (CDD). CDD decomposes an image into localized scale components; at each position, their amplitudes define a scale vector that describes how the measured intensity is distributed over spatial scale. We define the pixel-wise similarity $\Spix(x,y)$ as the normalized alignment of two local scale vectors. The normalization removes the overall amplitude, so $\Spix$ compares relative scale composition rather than absolute flux. We also define the scale-wise similarity $\Sscale(l)$ by comparing the two CDD component maps at each spatial scale. Spatial shifts are used to construct an empirical shifted reference distribution for $\Spix$. In Orion~A, the tracer with the highest similarity to the dust-derived column-density map changes from $^{12}$CO to $^{13}$CO to C$^{18}$O toward higher column density. In NGC~6334I(N), the line--continuum similarity decreases locally around the brightest compact structures, where radiative-transfer effects can alter the observed line morphology. In NGC~3627, CO is most similar to 21~$μ$m emission, and $\Sscale$ reaches its maximum at an intermediate sub-kpc scale. The method measures where two tracers have similar multiscale structure and at which scales their spatial distributions agree. The implementation is publicly available at https://github.com/meng-ke/Scale-Vector-Alignment.

astro-ph.IM↗
arXiv2026

Fast and accurate astronomical source deblending with Density-Peak Clustering

Source deblending is a fundamental challenge for current and forthcoming astronomical surveys, where increasing source density and image depth lead to a growing number of overlapping detections. Accurate deblending is essential for reliable measurements of source morphology and photometry, as well as for cosmological analyses. We present a redesign of the Advanced Density Peak (ADP) clustering algorithm, tailored to the identification and separation of blended astronomical sources within detection regions. We develop a validation framework combining realistic image simulations, automatically generated ground-truth segmentation, and label-invariant metrics. ADP is assessed against ASTErIsM, an established density-based astronomical deblender, using pairwise simulations, synthetic multi-source images, and Euclid Q1 public data. In pairwise simulations, the methods show comparable performance across a broad range of source separations and flux ratios, with photometric differences typically below 1% and reaching 5-7% in the most challenging cases, without systematic bias. In multi-source simulations, ADP recovers approximately 8% more ground-truth sources, while the positions of sources identified by both methods agree at the sub-pixel level. On Euclid Q1 public data, the methods show strong agreement in segmentation area, ellipticity, position angle, and photometry, with the largest differences for the smallest and faintest sources. ADP also provides a substantial computational advantage: end-to-end benchmarks on 19200 x 19200 pixel Euclid images require approximately 16-200 s, corresponding to speedups of 12-156x relative to ASTErIsM, with median and mean improvements of 36x and 54x, respectively. These results show that ADP provides scientifically competitive deblending at substantially lower computational cost, making it a promising approach for large-scale astronomical imaging surveys.

astro-ph.IM↗
arXiv2026

IceCube Upgrade status and perspectives

The IceCube Neutrino Observatory instruments one cubic kilometer of deep-glacial ice between 1450 m and 2450 m below the surface at the geographic South Pole to detect neutrinos via Cherenkov radiation of relativistic charged particles produced in their interactions. This detector is responsible for a number of key observations in neutrino astrophysics, which include the discovery of a high-energy astrophysical neutrino flux and, more recently, the galactic plane. During the austral summer of 2025/26, five new strings equipped with new photosensor designs were deployed as a dense infill in the middle of the existing detector. The science goals of this detector are twofold: Firstly, given the higher photocathode density, an improved atmospheric neutrino event selection and reconstruction at a few GeV can be achieved for enhanced capabilities to study neutrino oscillations. Secondly, novel calibration devices will improve the knowledge of the optical properties of the glacial ice and the detector response. These new calibration results will be applied to archival IceCube data, improving angular and spatial resolution of all detected astrophysical neutrino events. The IceCube Upgrade also serves as a first step towards the next-generation neutrino telescope at the South Pole, called IceCube-Gen2.

astro-ph.IM↗