Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset

Zhao Dong; Ka Chen; Zhaoyang Lv; Hong-Xing Yu; Yunzhi Zhang; Cheng Zhang; Yufeng Zhu; Stephen Tian; Zhengqin Li; Geordie Moffatt; Sean Christofferson; James Fort; Xiaqing Pan; Mingfei Yan; Jiajun Wu; Carl Yuheng Ren; Richard Newcombe

arXiv · 2504.08541

Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset

Zhao Dong·Ka Chen·Zhaoyang Lv·Hong-Xing Yu·Yunzhi Zhang·Cheng Zhang·Yufeng Zhu·Stephen Tian·Zhengqin Li·Geordie Moffatt·Sean Christofferson·James Fort·Xiaqing Pan·Mingfei Yan·Jiajun Wu·Carl Yuheng Ren·Richard Newcombe

View original source ↗Read source document ↗Export metadata ↓

Abstract

We introduce the Digital Twin Catalog (DTC), a new large-scale photorealistic 3D object digital twin dataset. A digital twin of a 3D object is a highly detailed, virtually indistinguishable representation of a physical object, accurately capturing its shape, appearance, physical properties, and other attributes. Recent advances in neural-based 3D reconstruction and inverse rendering have significantly improved the quality of 3D object reconstruction. Despite these advancements, there remains a lack of a large-scale, digital twin-quality real-world dataset and benchmark that can quantitatively assess and compare the performance of different reconstruction methods, as well as improve reconstruction quality through training or fine-tuning. Moreover, to democratize 3D digital twin creation, it is essential to integrate creation techniques with next-generation egocentric computing platforms, such as AR glasses. Currently, there is no dataset available to evaluate 3D object reconstruction using egocentric captured images. To address these gaps, the DTC dataset features 2,000 scanned digital twin-quality 3D objects, along with image sequences captured under different lighting conditions using DSLR cameras and egocentric AR glasses. This dataset establishes the first comprehensive real-world evaluation benchmark for 3D digital twin creation tasks, offering a robust foundation for comparing and improving existing reconstruction methods. The DTC dataset is already released at https://www.projectaria.com/datasets/dtc/ and we will also make the baseline evaluations open-source.

Explore related subjects

cs.GR cs.AI cs.CV cs.RO

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Zhao Dong, Ka Chen, Zhaoyang Lv, Hong-Xing Yu, Yunzhi Zhang, Cheng Zhang, Yufeng Zhu, Stephen Tian, Zhengqin Li, Geordie Moffatt, Sean Christofferson, James Fort, Xiaqing Pan, Mingfei Yan, Jiajun Wu, Carl Yuheng Ren, Richard Newcombe. 2025-05-18. Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset. https://arxiv.org/abs/2504.08541

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

Audio-driven facial animation is essential for immersive digital interaction, yet existing frameworks struggle to reconcile real-time streaming with high-fidelity personalization. Current methods either rely on latency-inducing audio look-ahead, or ask users to record scripted calibration sequences to pre-encode static identity embeddings that fail to capture dynamic idiosyncrasies. We present an end-to-end framework for personalized audio-driven facial motion generation, supporting causal, zero-lookahead streaming. We introduce two key innovations: (1) a causal multi-resolution motion tokenizer that captures both global temporal context and high-frequency articulatory details, and (2) a multi-modal style retriever that extracts stylistic priors from unstructured reference libraries by jointly querying ongoing audio and motion. Unlike prior retrieval mechanisms restricted to curated, fixed-size, or audio-only style banks, our design accepts arbitrary footage of the target identity, enabling personalization from a handful of casually recorded clips. By integrating these components, our method outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism, while preserving the streaming constraints of real-time telepresence. Code is available at https://github.com/xg-chu/Fallingwater.

cs.GR↗

arXiv2026

MultiCube: Compositional 3D Generation With Part-Level Semantic and Spatial Control

Digital 3D objects used in games and animation are often required to be compositional; that is, decomposed into semantically meaningful parts. Recent 3D generation methods can produce high-quality compositional objects conditioned on image or text prompts. Yet, such global conditioning lacks the precise part-level controllability required for professional creative workflows. To address this, we introduce MultiCube, a novel compositional 3D generation method that provides explicit, independent control over both the semantics and spatial arrangement of each part. MultiCube takes as input a global text prompt, a text schema specifying the desired parts, and a spatial layout indicating the bounding boxes of the parts in the given schema. It outputs a 3D object composed of distinct meshes, one per specified part, that adhere to the given semantic and spatial conditions. Our approach employs a two-stage diffusion process, first generating a schema- and layout-aligned monolithic mesh, then decomposing the mesh into individual parts simultaneously. A novel Part Layout Adapter is used to encode per-part conditions independently of the other parts. Experiments demonstrate that our method can generate high-quality compositional 3D objects with precise part-level control, including those with unique layouts difficult to achieve with text or image prompting alone. Project page: https://multi-cube.github.io

cs.GR↗

arXiv2026

FootprintRAG: Visual Analytics for Evidence Context Refinement in RAG-based Scientific Literature Exploration

Retrieval-Augmented Generation (RAG) is increasingly used to ground large language model (LLM) outputs in scientific literature. However, in open-ended literature exploration, the evidence context used for generation is often produced through hidden retrieval, reranking, assessment, and filtering steps. Users may receive retrieval summaries without knowing how the system constructed the evidence context, which evidence units were retained or discarded, or whether potentially useful evidence was excluded before synthesis. We present FootprintRAG, an LLM-agent-powered visual analytics system for evidence context refinement in RAG-based scientific literature exploration. The core idea is to treat the RAG evidence context as an explicit, inspectable, and revisable analytical object before generation. FootprintRAG parses scientific literature into text and figure evidence units, expands an initial query into parallel query variants, retrieves and assesses evidence across iterative rounds, and surfaces ERS-ranked supplementary candidates from the corpus-level evidence space. Through coordinated views, the system connects retrieval trajectories, evidence-state revision, and provenance-aware summary generation into a user-steerable workflow. We evaluate FootprintRAG through two case studies, a user study, and a workflow-level comparison with representative RAG systems. The results show that FootprintRAG helps users compare retrieval directions, revise candidate evidence, recover potentially overlooked evidence, and trace generated summaries back to supporting evidence units. FootprintRAG is available at https://github.com/meteorshowering/FootprintRAGVA.git.

cs.GR↗