GaussAnything: Semantic Intent-Driven Refinement of Evolving Gaussian Scenes for Standalone VR

Abstract

Deploying reconstructed 3D environments on standalone VR headsets are constrained by limited compute and memory, and conventional level-of-detail policies optimize for visibility without accounting for the user's explicit inspection intent. We present GaussAnything, a native OpenXR system for intent-conditioned reallocation and progressive publication of evolving semantic Gaussian+SDF scenes. GaussAnything resolves class- or instance-level queries to persistent 3D objects and reallocates a fixed Gaussian resident budget toward the selected object while retaining global context, applying incremental, stable-identity updates coordinated with the TSDF-derived mesh through a source-epoch mechanism. Across eight scenes, an object query concentrates 88-90% of the fixed client budget onto the queried object without enlarging it, on-device rendering reproduces the host render to within a small margin (up to 36.7 dB), and the standalone client renders each stereo frame at a steady-state GPU cost of roughly 10 ms within the frame budget of standard standalone panels.