$ϕ$-Scene: Physically Grounded Image-to-3D Scene Reconstruction

Abstract

Recent image-to-3D scene methods recover high-fidelity 3D objects with plausible arrangements, but often leave floatings and interpenetrations that limit physical validity and downstream use in interactive environments. We present $ϕ$-Scene, a physically grounded approach for open-vocabulary and compositional image-to-3D scene reconstruction that treats a scene not merely as a set of objects with predicted poses, but as a globally stable physical system. $ϕ$-Scene formulates reconstruction as topology-driven physical assembly: given an initial scene of reconstructed objects, it infers how objects support one another and settles them one by one in topological order. For each object, SDF-based optimization first resolves penetrations against the already-settled support context, and rigid-body simulation then settles the object into a stable equilibrium under real-world physical constraints. The resulting scene stays aligned to the reference image, with every object resting at a physically valid, stable contact configuration. On the 3D-Front benchmark, $ϕ$-Scene achieves the strongest overall performance among out-of-domain methods and remains highly competitive with in-domain baselines on standard reconstruction metrics. Human and MLLM studies prefer $ϕ$-Scene in visual quality, reference alignment, and physical plausibility. Dedicated physical metrics show that it substantially reduces penetration artifacts and yields much lower post-simulation drift. To our knowledge, $ϕ$-Scene is among the first image-to-3D scene reconstruction methods that explicitly reaches dynamic rigid-body equilibrium while preserving reference alignment.