Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography

Xinyi Wu; Haohong Wang; Aggelos K. Katsaggelos

Home/Archive/arXiv

arXiv · 2303.17041

Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography

Xinyi Wu·Haohong Wang·Aggelos K. Katsaggelos

View original source ↗Read source document ↗Export metadata ↓

Abstract

User-generated cinematic creations are gaining popularity as our daily entertainment, yet it is a challenge to master cinematography for producing immersive contents. Many existing automatic methods focus on roughly controlling predefined shot types or movement patterns, which struggle to engage viewers with the circumstances of the actor. Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects, considering frame aesthetics, spatial action, and emotional status in the 3D virtual stage. Following rule-of-thirds, our framework first modifies the initial camera placement to position the actor aesthetically. This adjustment is facilitated by a self-supervised adjustor that analyzes frame composition via camera projection. We then design a GAN model that can adversarially synthesize fine-grained camera movement based on the physical action and psychological state of the actor, using an encoder-decoder generator to map kinematics and emotional variables into camera trajectories. Moreover, we incorporate a regularizer to align the generated stylistic variances with specific emotional categories and intensities. The experimental results show that our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively. Live examples can be found in the supplementary video.

Explore related subjects

cs.MM cs.GR cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Xinyi Wu, Haohong Wang, Aggelos K. Katsaggelos. 2024-05-21. Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography. https://arxiv.org/abs/2303.17041

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs

Vision-language models are extending video understanding from offline clip analysis to continuous interactive streaming, but most research still emphasizes model capability rather than deployable low-latency interaction. This paper presents a unified edge-cloud system for real-time video VLM applications. Lightweight phone, smart glasses, PC, and pseudo-replay clients publish video and speech to a server runtime that provides shared ASR/TTS, session orchestration, backend adaptation, response delivery, and archive-backed measurement. The system integrates six representative video VLM backends with streaming or interaction-oriented capabilities and evaluates them across backend runtime, media transport, client-observed latency, and interaction behavior. With suitable backend selection and the WebRTC path, the tested system reaches approximately 0.9 to 1.0 s to first VLM text and 1.3 to 1.5 s to first non-silent TTS audio, while exposing backend adaptation costs and differences in real-time interaction behavior.

cs.MM↗

arXiv2026

MotionCanvas: Learning Implicit Motion Planning from Composable Kinematic Cues

Professional character animation requires both natural motion and precise, versatile control. For example, it is common for the creators to define the timing of a specified action, to control the motion range of the character's arm swing, and the route the character walks through, like specifying various kinematic motion cues on a ``motion canvas''. This motivates us to propose MotionCanvas, a model that supports \emph{cue-conditioned implicit motion planning} to faithfully and coherently connect all cues, dense or sparse, full or partial, into one full-body motion sequence. Specifically, MotionCanvas represents heterogeneous kinematic cues on a shared motion canvas, where position and rotation values are specified across body joints and time. A shared flow-matching model generates motion conditioned on this canvas, with optional language and input motion; cue imputation keeps the specified canvas values fixed in both training and sampling. To learn coherent completion across different cue sets, we train with a compositional cue sampler that varies when cues are applied, which positions or rotations are specified, and how they are combined. Together, these designs enable a single generator to synthesize globally coherent actions that jointly satisfy compatible heterogeneous cues. We test this planning ability with temporal, root, and body-part cues---alone and in combination---and language-guided editing. We naturally extend this evaluation to sequential generation and motion repair, since both require the same ability to organize coherent motion from kinematic cues. Across these evaluations, MotionCanvas establishes state-of-the-art results in controlled-motion quality, mixed-cue adherence, sequential generation, instruction editing, and motion repair while preserving its text-to-motion capability.

cs.MM↗

arXiv2026

Multi-Faceted Evaluation and Mitigation of Emotion Hallucinations in MLLMs

Multimodal large language models (MLLMs) have shown strong potential in open-ended emotion understanding, yet they often generate emotion hallucinations. Evaluating such hallucinations is particularly challenging for two reasons. First, emotion understanding spans multiple cognitive facets, from multimodal perception to psychological reasoning. Second, emotional interpretations are expressed in free-form language, making existing closed-ended protocols insufficient for evaluation. To address these challenges, we introduce EHR (Emotion Hallucination Rate), an evaluator that quantifies emotion hallucinations across six facets: expression, action, audio, instinct, logic, and conclusion. Using EHR, we reveal that existing mitigation methods often reduce hallucinations in some facets while aggravating them in others, exposing the limitation of coarse-grained correction and the need for facet-aware localization and mitigation. Motivated by this finding, we propose HMER (Hallucination-aware Memory-guided Emotion Reasoning), a training-free framework for emotion hallucination mitigation. HMER maintains a Hallucination Memory that records localized hallucinated claims and enables targeted logit rectification, together with an Anchor Memory that preserves reliable intermediate reasoning states to stabilize subsequent generation. By selectively suppressing unreliable cues while preserving trustworthy reasoning context, HMER enables fine-grained mitigation across diverse hallucination facets. Extensive experiments on 19 MLLMs demonstrate the prevalence of emotion hallucinations and the effectiveness of our framework across diverse model architectures.

cs.MM↗