Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2504.10931

A Highly Efficient Cross-matching Scheme using Learned Index Structure

Phu-Minh Lam·Dongwei Fan·Hongbo Wei·Jun Wang·Yu Zhou·Qi Ma·Baolong Zhang·Xiazhao Zhang·Yongheng Wang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Spatial data fusion is a bottleneck when it meets the scale of 10 billion records. Cross-matching celestial catalogs is just one example of this. To challenge this, we present a framework that enables efficient cross-matching using Learned Index Structures. Our approach involves a data transformation method to map multi-dimensional data into easily learnable distributions, coupled with a novel search algorithm that leverages the advantages of model pairs, significantly enhancing the efficiency of nearest-neighbor search. In this study, we utilized celestial catalog data derived from astronomical surveys to construct the index and evaluated the speed of the cross-matching process. Using the HEALPix segmentation scheme, we built an independent model object for each tile and developed an end-to-end pipeline to construct a framework with semantic guarantees for record retrieval in query and range search. Our results show that the proposed method improves cross-matching speed by more than four times compared to KD-trees for a radius range between 1 milli-arcseconds and 100 arcseconds.

Explore related subjects

astro-ph.IM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Phu-Minh Lam, Dongwei Fan, Hongbo Wei, Jun Wang, Yu Zhou, Qi Ma, Baolong Zhang, Xiazhao Zhang, Yongheng Wang. 2025-04-15. A Highly Efficient Cross-matching Scheme using Learned Index Structure. https://arxiv.org/abs/2504.10931

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Radial Pulsations in Polaris: A Secondary Science Application of Cherenkov Telescopes via Intensity Interferometry

Ground-based Cherenkov telescopes, which are typically inoperative during moonlit nights for gamma-ray observations, offer a valuable opportunity during this time for secondary scientific applications through Intensity Interferometry (II). Recent developments and observations suggest that implementing II instrumentation on existing and planned Imaging Atmospheric Cherenkov Telescopes (IACTs) can significantly advance optical stellar measurements. Motivated by the resurgence of II efforts over the past two decades, this work presents simulations demonstrating the estimation of stellar parameters for a radially pulsating star, such as Polaris, using either a single telescope or multiple telescopes. For single-telescope simulations, we assume that the photon pixels in the camera are mapped onto four distinct regions of the aperture, generating multiple baselines and enabling enhanced observational plane coverage. These results highlight the potential of Cherenkov telescopes in India for high-resolution optical astronomy during otherwise inoperative periods and offer promising insights into the characterization of bright stellar objects with unprecedented precision.

astro-ph.IM↗
arXiv2026

Parallel adaptive reweighting importance sampling for Bayesian astrophysics

Efficient sampling from high-dimensional, multi-modal posteriors is a central challenge in Bayesian inference across the physical sciences. Gravitational-wave (GW) astronomy, where likelihood evaluations are computationally expensive and posteriors are complex, is one application motivating this work. Popular families of methods like Markov-chain Monte Carlo, nested sampling, and importance sampling all rely on proposal distributions to guide exploration. Because prior knowledge of the target is often limited, practitioners can adopt adaptive proposals that iteratively refine themselves using information gained from previously drawn samples. Traditional adaptive strategies, however, struggle in high-dimensional multi-modal settings: complex, non-linear correlations are hard to capture, and hyperparameters typically require tedious, problem-specific tuning. To address these issues, we introduce Parallel Adaptive Reweighting Importance Sampling (PARIS; descriptively, ``seed sampling''). PARIS models its proposal as a Gaussian mixture whose component centers are the existing samples and whose component weights match the current importance weights. New draws from the proposal therefore concentrate around high-weight regions, while candidate points in unexplored areas receive intentionally inflated weights. As the algorithm continuously reweights all samples up to the latest proposal, any initial over-weighting self-corrects once additional neighbor samples are collected. To enable rapid reweighting, we present an efficient update scheme and evaluate PARIS on illustrative toy problems and more realistic gravitational-wave parameter estimation tasks. PARIS achieves accurate posterior reconstruction and evidence estimation with substantially fewer function evaluations than competing approaches, highlighting its promise for widespread use in astrophysical data analysis.

astro-ph.IM↗
arXiv2026

Ising noise filter: physics-informed filtering for particle detectors

We present the Ising noise filter, a highly portable, graph-based pre-filtering algorithm for early-stage background suppression in particle accelerators and astrophysical detectors. Standard noise rejection methods relying on track fitting suffer from severe combinatorial explosion. Our method bypasses this by mapping individual detector hits to a network of binary spins and minimizing an energy functional. The interaction kernels are physics-informed, tailored to the underlying physics and geometry of the experiment. We demonstrate the efficacy of this approach in two distinct experimental regimes. Applied to the Baikal-GVD neutrino telescope the filter yields fast, standard-quality noise rejection with 96.8% recall for astrophysical neutrinos. For the SPD detector at the NICA collider the filter attains recall of 97% on a toy Monte Carlo sample. Furthermore, when combined with a Peterson--Hopfield network for track finding, our physics-informed coupling improves the TrackML score from 0.5 to 0.95.

astro-ph.IM↗