Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2012.08455

Scaling pair count to next galaxy surveys

S. Plaszczynski·J. E. Campagne·J. Peloton·C. Arnault
View original source ↗Read source document ↗Export metadata ↓

Abstract

Counting pairs of galaxies or stars according to their distance is at the core of real-space correlation analyzes performed in astrophysics and cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of billions of galaxies challenging our ability to perform such counting in a minute-scale time relevant for the usage of simulations. The problem is only limited by efficient access to the data, hence belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of nonhierarchical sphere pixelization based on cube symmetries and develop a new one dubbed the "Similar Radius Sphere Pixelization" (SARSPix) with very close to square pixels. It provides the most adapted indexing over the sphere for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In each case we achieve the construction of a standard pair-distance histogram in about 2 minutes, using a simple algorithm that is shown to scale, over a moderate number of nodes (16 to 64). This illustrates the potential of this new techniques in the field of astronomy where data access is becoming the main bottleneck. They can be easily adapted to other use-cases as nearest-neighbors search, catalog cross-match or cluster finding. The software is publicly available from https://github.com/astrolabsoftware/SparkCorr.

Explore related subjects

astro-ph.IMastro-ph.CO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

S. Plaszczynski, J. E. Campagne, J. Peloton, C. Arnault. 2022-01-03. Scaling pair count to next galaxy surveys. https://doi.org/10.1093/mnras%2Fstab3640

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Scale-Vector Alignment: A Scale-Aware Framework for Spatially Resolved Morphological Similarity in Astronomical Images

Astronomical maps made with different tracers are not expected to have identical morphology. Excitation, optical depth, chemistry, radiation, and ISM phase alter the response of a tracer, and the resulting differences can depend on both position and spatial scale. We propose scale-vector alignment, a scale-aware method based on Constrained Diffusion Decomposition (CDD). CDD decomposes an image into localized scale components; at each position, their amplitudes define a scale vector that describes how the measured intensity is distributed over spatial scale. We define the pixel-wise similarity $\Spix(x,y)$ as the normalized alignment of two local scale vectors. The normalization removes the overall amplitude, so $\Spix$ compares relative scale composition rather than absolute flux. We also define the scale-wise similarity $\Sscale(l)$ by comparing the two CDD component maps at each spatial scale. Spatial shifts are used to construct an empirical shifted reference distribution for $\Spix$. In Orion~A, the tracer with the highest similarity to the dust-derived column-density map changes from $^{12}$CO to $^{13}$CO to C$^{18}$O toward higher column density. In NGC~6334I(N), the line--continuum similarity decreases locally around the brightest compact structures, where radiative-transfer effects can alter the observed line morphology. In NGC~3627, CO is most similar to 21~$μ$m emission, and $\Sscale$ reaches its maximum at an intermediate sub-kpc scale. The method measures where two tracers have similar multiscale structure and at which scales their spatial distributions agree. The implementation is publicly available at https://github.com/meng-ke/Scale-Vector-Alignment.

astro-ph.IM↗
arXiv2026

Fast and accurate astronomical source deblending with Density-Peak Clustering

Source deblending is a fundamental challenge for current and forthcoming astronomical surveys, where increasing source density and image depth lead to a growing number of overlapping detections. Accurate deblending is essential for reliable measurements of source morphology and photometry, as well as for cosmological analyses. We present a redesign of the Advanced Density Peak (ADP) clustering algorithm, tailored to the identification and separation of blended astronomical sources within detection regions. We develop a validation framework combining realistic image simulations, automatically generated ground-truth segmentation, and label-invariant metrics. ADP is assessed against ASTErIsM, an established density-based astronomical deblender, using pairwise simulations, synthetic multi-source images, and Euclid Q1 public data. In pairwise simulations, the methods show comparable performance across a broad range of source separations and flux ratios, with photometric differences typically below 1% and reaching 5-7% in the most challenging cases, without systematic bias. In multi-source simulations, ADP recovers approximately 8% more ground-truth sources, while the positions of sources identified by both methods agree at the sub-pixel level. On Euclid Q1 public data, the methods show strong agreement in segmentation area, ellipticity, position angle, and photometry, with the largest differences for the smallest and faintest sources. ADP also provides a substantial computational advantage: end-to-end benchmarks on 19200 x 19200 pixel Euclid images require approximately 16-200 s, corresponding to speedups of 12-156x relative to ASTErIsM, with median and mean improvements of 36x and 54x, respectively. These results show that ADP provides scientifically competitive deblending at substantially lower computational cost, making it a promising approach for large-scale astronomical imaging surveys.

astro-ph.IM↗
arXiv2026

IceCube Upgrade status and perspectives

The IceCube Neutrino Observatory instruments one cubic kilometer of deep-glacial ice between 1450 m and 2450 m below the surface at the geographic South Pole to detect neutrinos via Cherenkov radiation of relativistic charged particles produced in their interactions. This detector is responsible for a number of key observations in neutrino astrophysics, which include the discovery of a high-energy astrophysical neutrino flux and, more recently, the galactic plane. During the austral summer of 2025/26, five new strings equipped with new photosensor designs were deployed as a dense infill in the middle of the existing detector. The science goals of this detector are twofold: Firstly, given the higher photocathode density, an improved atmospheric neutrino event selection and reconstruction at a few GeV can be achieved for enhanced capabilities to study neutrino oscillations. Secondly, novel calibration devices will improve the knowledge of the optical properties of the glacial ice and the detector response. These new calibration results will be applied to archival IceCube data, improving angular and spatial resolution of all detected astrophysical neutrino events. The IceCube Upgrade also serves as a first step towards the next-generation neutrino telescope at the South Pole, called IceCube-Gen2.

astro-ph.IM↗