MoSAT: Human Motion Generation from Spatial Audio and Textual Description

Shuyang Xu; Zhiyang Dou; Yiduo Hao; Zekun Li; Liang Pan; Jingbo Wang; Cheng Lin; Yuan Liu; Wenping Wang; Mingmin Zhao; Taku Komura

arXiv · 2609.23797

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

Shuyang Xu·Zhiyang Dou·Yiduo Hao·Zekun Li·Liang Pan·Jingbo Wang·Cheng Lin·Yuan Liu·Wenping Wang·Mingmin Zhao·Taku Komura

View original source ↗Read source document ↗Export metadata ↓

Abstract

Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.

Explore related subjects

cs.GR cs.CV cs.RO cs.SD

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Shuyang Xu, Zhiyang Dou, Yiduo Hao, Zekun Li, Liang Pan, Jingbo Wang, Cheng Lin, Yuan Liu, Wenping Wang, Mingmin Zhao, Taku Komura. 2026-09-20. MoSAT: Human Motion Generation from Spatial Audio and Textual Description. https://arxiv.org/abs/2609.23797

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Constrained Program Generation for 3D Reaction Animation with a 0.8B Model

Visualizing a chemical reaction requires making its molecular changes visible while keeping the animation faithful to the stated chemistry. Equations, structural diagrams and molecular viewers provide complementary descriptions, but assembling an interactive three-dimensional explanation still requires specifying the changes and checking their consistency. We present ChemXRG, a domain-specific language (DSL) framework that addresses this gap by representing a reaction animation as an executable program. Persistent atom identifiers and explicit bond, charge and grouping operations connect the symbolic reaction to the displayed transformation. This shared representation lets generation, verification and rendering operate on the same account of what changes. Given known, atom-mapped reactant and product structures, reaction-grounded constraints fix input-determined facts and restrict action choices; execution checks validate the resulting transformation before geometry and frames are constructed. We implement this paradigm with a reaction-program corpus and ChemQwen, a trained 0.8B DSL generator. Paired and component evaluations show improved compiler acceptance and normalized full-program agreement under input-conditioned constraints, while identifying remaining failures that require execution checks. A public browser application demonstrates the connection from symbolic reaction descriptions to inspectable programs and interactive 3D animations.

cs.GR↗

arXiv2026

NaRPA: Navigation and Rendering Pipeline for Astronautics

This paper presents the applications of scientific ray-tracing in modeling and simulating light transport for space-borne image data generation. A ray-tracing engine, the Navigation and Rendering Pipeline for Astronautics (NaRPA), is introduced as a rendering framework to generate virtual datasets and support simulations for robust navigation pipelines. Sensor and environment models that enable the synthesis of space-to-space and ground-to-space virtual observations are presented. The work demonstrates the capabilities of simulating passive and active vision-based sensors using NaRPA to facilitate the design, testing, and verification of aerospace visual navigation algorithms. Additionally, the paper describes a velocimeter LiDAR model and its statistical validation with experimental data.

cs.GR↗

arXiv2026

VoroUDF: Meshing Unsigned Distance Fields with Voronoi Optimization

We present VoroUDF, an algorithm for reconstructing high-quality triangle meshes from Unsigned Distance Fields (UDFs). Our algorithm supports non-manifold geometry, sharp features, and open boundaries, without relying on error-prone inside/outside estimation, restrictive look-up tables nor topologically noisy optimization. Unlike fixed-grid approaches, our Voronoi-based formulation optimizes a set of movable seeds that travel freely along the iso-surface, jointly minimizing a tangent-plane fitting energy, driven by a fixed set of surface samples queried once from the UDF and its gradient, and a repulsion energy that keeps the seeds evenly distributed. It achieves significantly improved topological consistency and geometric fidelity compared to existing methods, while producing lightweight meshes suitable for downstream real-time and interactive applications.

cs.GR↗