Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2506.20010

Near Data Processing in Taurus Database

Shu Lin·Arunprasad P. Marathe·Per-Ȧke Larson·Chong Chen·Calvin Sun·Paul Lee·Weidong Yu
View original source ↗Read source document ↗Export metadata ↓

Abstract

Huawei's cloud-native database system GaussDB for MySQL (also known as Taurus) stores data in a separate storage layer consisting of a pool of storage servers. Each server has considerable compute power making it possible to push data reduction operations (selection, projection, and aggregation) close to storage. This paper describes the design and implementation of near data processing (NDP) in Taurus. NDP has several benefits: it reduces the amount of data shipped over the network; frees up CPU capacity in the compute layer; and reduces query run time, thereby enabling higher system throughput. Experiments with the TPCH benchmark (100 GB) showed that 18 out of 22 queries benefited from NDP; data shipped was reduced by 63 percent; and CPU time by 50 percent. On Q15 the impact was even higher: data shipped was reduced by 98 percent; CPU time by 91 percent; and run time by 80 percent.

Explore related subjects

cs.DB

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Shu Lin, Arunprasad P. Marathe, Per-Ȧke Larson, Chong Chen, Calvin Sun, Paul Lee, Weidong Yu. 2025-06-24. Near Data Processing in Taurus Database. https://doi.org/10.1109/icde53745.2022.00170

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

MLSkip: Data Skipping for ML Filters via Lightweight Metadata

Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.

cs.DB↗
arXiv2026

IBLTs Measure Before They Decode: Self-Sizing Set Reconciliation for Database Consistency Verification

Cross-system data replication pipelines cannot confirm end-to-end consistency from the local guarantees of each hop, so the two endpoints must be compared directly on a periodic basis. Once the rows of a fixed snapshot are normalized into fingerprints, the task reduces to finding the symmetric difference of the two sets. An Invertible Bloom Lookup Table (IBLT) reconciles the sets with communication that grows only with the difference cardinality $d$, independent of table size, but its capacity must be fixed while $d$ is still unknown. Across 41,603 production reconciliations over 90 days, nonzero $d$ spans about seven orders of magnitude, and no reliable empirical constant exists. We show that the count array of an IBLT has already measured $d$ before decoding. The measurement is in-band: it is carried by the recovery sketch itself and adds no bytes dedicated to estimation. A mapping-aware theorem extends the construction to Irregular, Rateless, and MET IBLTs. The protocol reads the estimate only after a decoding failure; we prove that the failure-conditioned lower quantile bounds the risk of underestimation, which gives the second-round capacity a configurable success-probability guarantee. The resulting self-sizing protocol attempts recovery with a small first-round sketch and stops on success; on failure it reads $d$, sizes the second round, and completes reconciliation in at most two rounds. Against a controlled oracle, communication is 1.29-1.47 times that of a scheme given $d$ in advance. Production workload characterization, relational-database replay, and a cross-city KV deployment confirm the end-to-end mechanism. In production on an Oracle-MySQL link, all completed runs succeeded within two rounds, 90.7% on the 1-RTT fast path with a single 16 KB sketch.

cs.DB↗
arXiv2026

Towards Effective Orchestration of AI x DB Workloads

AI-driven analytics are increasingly crucial to data-centric decision-making. Executing relational and AI operators in separate runtimes prevents the database optimizer and runtime from coordinating operator ordering, model placement, batching, and state reuse. Integrating AI operators into database engines enables such coordination but raises challenges in jointly optimizing query processing and model execution, scheduling under resource contention, and reusing relational intermediates and AI artifacts. This paper formalizes AIxDB workloads as iterative, concurrent, and shareable executions that interleave relational and AI operators. We then advocate database-native orchestration as a paradigm for redesigning database engines for these workloads and distill two design principles: holistic AIxDB co-optimization and unified AIxDB cache management. We present NeurEngine as a proof-of-concept prototype and report preliminary results illustrating the performance benefits of database-native orchestration

cs.DB↗