Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.32247

MetaDB: Putting File-System Structure Back into Distributed Databases

Zebin Ren·Biao Cao·Kang Chen·Zhihui Yin·Qiushi Chen·Jielong Jian·Jiwei Xiong·Xinxing Wang·Yan Xing·Liguo Duan·Ran Zheng·Xianjun Meng·Zhenyu Hou
View original source ↗Read source document ↗Export metadata ↓

Abstract

Storage systems in production increasingly rely on distributed databases for metadata management, yet our experience at Baidu shows that treating a general-purpose database as a black box leaves substantial performance and efficiency untapped. File-system metadata exhibits directory-tree locality, differentiated consistency requirements, and clustered deletions that general-purpose databases largely ignore. We thus propose MetaDB, a metadata-aware distributed transactional database that serves as the shared metadata layer for Baidu's large-scale storage systems. MetaDB addresses three key mismatches between general-purpose databases and metadata workloads: Data Model Mismatch, Transaction Model Mismatch, and Storage Implementation Mismatch. To bridge these gaps, MetaDB combines (1) tree-topology-induced sharding and execution, (2) semantic-aware off-critical-path processing, and (3) a lifecycle-aware storage engine with in-memory MVCC and locality-aware compaction. MetaDB achieves 2.2x-4.1x the throughput of our previous general-purpose database and halves the number of metadata machines. Deployed in production at Baidu for over five years, MetaDB unifies metadata management across the company's storage fleet and scales to over 1,000 metadata nodes, tens of trillions of metadata records, and exabyte-scale storage in a single cluster.

Explore related subjects

cs.DBcs.DC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zebin Ren, Biao Cao, Kang Chen, Zhihui Yin, Qiushi Chen, Jielong Jian, Jiwei Xiong, Xinxing Wang, Yan Xing, Liguo Duan, Ran Zheng, Xianjun Meng, Zhenyu Hou. 2026-10-23. MetaDB: Putting File-System Structure Back into Distributed Databases. https://arxiv.org/abs/2610.32247

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Scaling Graph Query Processing: Focused Exploration on Partitions Using Load Balancing

The focus of this paper is on query processing and not search (approximate or exact). The goal is to find all or k answers and the expressiveness of a query (as opposed to a search expression which is typically simpler) is considered important. Although SQL query processing has matured over the last several decades, query processing over graph data has not received the same attention. Only recently, systems, such as Neo4J, and others have started using graphs for data representation on a large scale. Their approaches and algorithms remain proprietary. In this paper, we address the problem of answering graph queries and their efficient processing on arbitrary-sized graphs. We propose the partitioning of graphs and the use of the distributed paradigm to achieve efficiency and scalability. Our approach takes a subgraph as query input including wild cards and comparison operations. Our partitioned approach leverages focused exploration and distributed processing to achieve the desired parallelism and improve the response time. We also present heuristics to reduce the number of partitions loaded for query processing. We present our partitioning algorithm, distributed processing architecture used, and load balancing for computing query answers. We establish the correctness of the approach in the presence of partitions, compare heuristics for load balancing, and extensive experimental results over large synthetic and real-world graph data sets.

cs.DB↗
arXiv2026

OCTOPUS: Towards A Tag Model for Cross-Query Reuse in Semantic Query Processing

Large language models make it possible to pose semantic questions over diverse, multi-modal data. Recent systems give such workloads the declarative feel of a database through semantic operators: relational operators whose conditions a model judges. Most of them now run on a relational engine, yet almost none keeps what the model says past the query: every judgment is discarded on completion and paid for again the next time a query needs it. We present the tag model and tag operators, the right unit for semantic processing, and OCTOPUS, a prototype built as a middleware over DBMS. Two mechanisms are central. First, the tag: each semantic answer a model produces is written back to the relation as data, which the database engine filters, indexes, and stores as it would any other. Second, the tag operator: the unit into which common semantic operators are translated, which fixes the granularity of reuse. A model judges a tuple once, and later queries reuse that judgment as data: through relationships between predicates, a stored judgment can accept a tuple for a later predicate, eliminate one that cannot satisfy it, simplify its test, or order the search of a query that stops early. On the movie scenario of SemBench, our prototype sends the model 10 to 108x fewer rows than iPDB, ThalamusDB, Palimpzest, and LOTUS at broadly competitive quality, and three of its eight queries issue no model call at all. More broadly, a database that keeps its judgments lets its schema grow from the questions its users ask, so that each semantic answer is paid for once, stays consistent across queries, and can be audited through its provenance.

cs.DB↗
arXiv2026

Tile-Based Decompression in Compiled GPU Engines

Modern GPUs reach record analytical performance only when a query's working set fits within its GPU memory budget, and that budget is under growing pressure: GPUs are partitioned and shared, working sets are outgrowing high-bandwidth memory (HBM), and HBM is a fixed on-device resource whose per-dollar capacity has stagnated across generations. Lightweight compression can relieve this pressure by shrinking both footprint and bandwidth demand, but existing GPU compressed-execution techniques either bind decompression to a specific execution model or materialize decompressed columns in HBM before query execution. We present a compression subsystem for compiled GPU query engines that treats decompression as a first-class physical operator. Compression-specific logic is isolated behind a narrow backend interface, so new GPU-oriented encodings can be added without rewriting downstream operators. We implement the design in Proteus and study where decompressed values should be materialized in the GPU memory hierarchy. Our compiler-integrated shared-memory strategy fuses cooperative tile decompression with tuple-at-a-time query execution inside a single generated GPU kernel, avoiding decompressed-column materialization in HBM. Compared with HeavyDB and Sirius on GPU-resident data, fused on-chip decompression achieves competitive query times while reducing the resident memory footprint by up to 4x and memory-bandwidth consumption by up to 6x.

cs.DB↗