arXiv subjects

Weihan Xu

Publications and source records attributed to Weihan Xu.

14 recordsLinked to original sources

arXiv2026

Clips2Story: Training-Free Video Storyboarding and Editing Using Multimodal Retrieval-Embedded Generation

Video editors often work with large collections of fragmented footage and must select and organize relevant clips into a coherent story. However, many existing video editing and summarization methods assume a temporally ordered source video and aim to preserve its original narrative and temporal structure. In this paper, we introduce Clips2Story, a training-free, LLM-based pipeline that enables users to create keyword-specified short videos from a pool of video clips. Given a pool of clips and a user-specified keyword, we first extract text-based multimodal descriptors for each clip and retrieve the clips most relevant to the keyword. We then provide the textual descriptors of these candidate clips to an LLM, which selects and arranges them into an editable storyboard with connective narrations. Finally, we assemble the selected clips and retrieve accompanying visuals for the narrations to compose the output video. To evaluate the proposed pipeline, we construct a synthetic evaluation dataset from 100 long-form videos across five genres by segmenting each video into short clips using a scene-change detection model and shuffling the resulting clips to remove their temporal order. Through objective evaluation and qualitative examples, we demonstrate that Clips2Story can effectively retrieve keyword-relevant clips and compose coherent storyboards based on user-specified keywords. Human evaluation further shows that Clips2Story outperforms a strong baseline in terms of narrative coherence, keyword alignment, and overall realism.

cs.CV↗

arXiv2026

ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.

cs.RO↗

arXiv2026

CLEARMIND: Closed-Form Laplacian Embeddings of Arakelov-Green Resistance for Morphological Intrinsic Neuronal Distances

Tropical geometry turns a metric graph into a flat torus, its tropical Jacobian, and measures distances between the images of its points by a closest vector problem on the period lattice. We prove that for points of the graph this problem is solved by geodesics: the squared tropical polarization distance equals the path metric minus the Arakelov-Green (AG) distance, an effective-resistance kernel given in closed form by the Laplacian pseudoinverse. Equivalently, the AG distance and the path metric are the minimal energies of real and of integral unit flows, and the squared tropical distance is exactly their integrality gap. The full tropical distance matrix is thus computable in cubic time. On this identity we build CLEARMIND, a training-free descriptor of 3D neuronal morphology. A reconstruction is reduced to its branching skeleton, neurite tips near the soma are joined to it, short bridges are contracted, and the AG matrix of the result is summarized by its spectrum. Every step has an exact algebraic description; for instance, the period matrix records the shared soma-to-tip path lengths, and the spectrum has exactly one positive eigenvalue, so its absolute values determine it. The signature needs no lattice search, is invariant to vertex order, rigid motions and subdivision, and is Lipschitz in the edge lengths. On ACT-4, JML-4 and BIL-6, appending the 64-dimensional spectrum improves a point-cloud GNN and MorphVAE on every dataset, by up to 20.3 points; a Tree-LSTM with the spectrum outperforms every reimplemented baseline; and an MLP on the spectrum alone is a competitive classifier that surpasses every reimplemented deep baseline on ACT-4. On BREC, a GIN with AG eigenvector encodings distinguishes 70.0% of the pairs beyond the 1-WL limit, including 33% of the CFI pairs, more than PPGN (23%) and I$^2$-GNN (21%), and the spectrum alone separates 217 of 400 pairs without training.

cs.LG↗

arXiv2026

AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes

Adding or removing a sounding object requires coordinated changes to visual content and sound while preserving the surrounding scene. Yet paired supervision for localized non-speech audiovisual editing remains limited, as visual and acoustic edits must target the same object and isolate its sound from overlapping sources. To address this gap, we introduce \textit{AVIOBench}, a dataset comprising 37.9 hours of paired audiovisual examples spanning 1{,}878 target-object names. AVIOBench links the visual presence and acoustic contribution of each target object through a shared identity and visual mask. Our automated pipeline uses visual grounding and cross-modal consistency to select target-sound removal candidates, then jointly refines the audiovisual pairs to improve perceptual quality and cross-modal consistency. Building on this dataset, we propose \textit{AVIO}, which adapts a pretrained text-to audiovisual generation model through source-conditioned feature modulation to jointly learn object addition and removal. A reference-frame curriculum gradually reduces reference conditioning during training, enabling one model to perform instruction-only editing with optional visual guidance. Quantitative and qualitative evaluations demonstrate effective audiovisual object removal and addition, with optional reference guidance providing appearance and placement control for addition.

cs.AI↗

arXiv2025

Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal

Joint editing of audio and visual content is crucial for precise and controllable content creation. This new task poses challenges due to the limitations of paired audio-visual data before and after targeted edits, and the heterogeneity across modalities. To address the data and modeling challenges in joint audio-visual editing, we introduce SAVEBench, a paired audiovisual dataset with text and mask conditions to enable object-grounded source-to-target learning. With SAVEBench, we train the Schrodinger Audio-Visual Editor (SAVE), an end-to-end flow-matching model that edits audio and video in parallel while keeping them aligned throughout processing. SAVE incorporates a Schrodinger Bridge that learns a direct transport from source to target audiovisual mixtures. Our evaluation demonstrates that the proposed SAVE model is able to remove the target objects in audio and visual content while preserving the remaining content, with stronger temporal synchronization and audiovisual semantic correspondence compared with pairwise combinations of an audio editor and a video editor.

cs.CV↗

arXiv2025

Video-Guided Text-to-Music Generation Using Public Domain Movie Collections

Despite recent advancements in music generation systems, their application in film production remains limited, as they struggle to capture the nuances of real-world filmmaking, where filmmakers consider multiple factors-such as visual content, dialogue, and emotional tone-when selecting or composing music for a scene. This limitation primarily stems from the absence of comprehensive datasets that integrate these elements. To address this gap, we introduce Open Screen Soundtrack Library (OSSL), a dataset consisting of movie clips from public domain films, totaling approximately 36.5 hours, paired with high-quality soundtracks and human-annotated mood information. To demonstrate the effectiveness of our dataset in improving the performance of pre-trained models on film music generation tasks, we introduce a new video adapter that enhances an autoregressive transformer-based text-to-music model by adding video-based conditioning. Our experimental results demonstrate that our proposed approach effectively enhances MusicGen-Medium in terms of both objective measures of distributional and paired fidelity, and subjective compatibility in mood and genre. To facilitate reproducibility and foster future work, we publicly release the dataset, code, and demo.

cs.SD↗

arXiv2025

Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset

Recent years have seen many audio-domain text-to-music generation models that rely on large amounts of text-audio pairs for training. However, symbolic-domain controllable music generation has lagged behind partly due to the lack of a large-scale symbolic music dataset with extensive metadata and captions. In this work, we present MetaScore, a new dataset consisting of 963K musical scores paired with rich metadata, including free-form user-annotated tags, collected from an online music forum. To approach text-to-music generation, We employ a pretrained large language model (LLM) to generate pseudo-natural language captions for music from its metadata tags. With the LLM-enhanced MetaScore, we train a text-conditioned music generation model that learns to generate symbolic music from the pseudo captions, allowing control of instruments, genre, composer, complexity and other free-form music descriptors. In addition, we train a tag-conditioned system that supports a predefined set of tags available in MetaScore. Our experimental results show that both the proposed text-to-music and tags-to-music models outperform a baseline text-to-music model in a listening test. While a concurrent work Text2MIDI also supports free-form text input, our models achieve comparable performance. Moreover, the text-to-music system offers a more natural interface than the tags-to-music model, as it allows users to provide free-form natural language prompts.

cs.SD↗

arXiv2025

REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing

Short videos are an effective tool for promoting contents and improving knowledge accessibility. While existing extractive video summarization methods struggle to produce a coherent narrative, existing abstractive methods cannot `quote' from the input videos, i.e., inserting short video clips in their outputs. In this work, we explore novel video editing models for generating shorts that feature a coherent narrative with embedded video insertions extracted from a long input video. We propose a novel retrieval-embedded generation framework that allows a large language model to quote multimodal resources while maintaining a coherent narrative. Our proposed REGen system first generates the output story script with quote placeholders using a finetuned large language model, and then uses a novel retrieval model to replace the quote placeholders by selecting a video clip that best supports the narrative from a pool of candidate quotable video clips. We examine the proposed method on the task of documentary teaser generation, where short interview insertions are commonly used to support the narrative of a documentary. Our objective evaluations show that the proposed method can effectively insert short video clips while maintaining a coherent narrative. In a subjective survey, we show that our proposed method outperforms existing abstractive and extractive approaches in terms of coherence, alignment, and realism in teaser generation.

cs.CV↗

arXiv2024

Recurrent Neural Network on PICTURE Model

Intensive Care Units (ICUs) provide critical care and life support for most severely ill and injured patients in the hospital. With the need for ICUs growing rapidly and unprecedentedly, especially during COVID-19, accurately identifying the most critical patients helps hospitals to allocate resources more efficiently and save more lives. The Predicting Intensive Care Transfers and Other Unforeseen Events (PICTURE) model predicts patient deterioration by separating those at high risk for imminent intensive care unit transfer, respiratory failure, or death from those at lower risk. This study aims to implement a deep learning model to benchmark the performance from the XGBoost model, an existing model which has competitive results on prediction.

cs.AI↗

arXiv2024

TeaserGen: Generating Teasers for Long Documentaries

Teasers are an effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling on the input videos, while necessitating maintaining audiovisual alignments, managing scene changes and preserving factual accuracy for the output teasers. Due to the lack of a publicly-available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, a collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose a new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration of the documentary using a pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-vision models. For narration-video matching, we explore two approaches: a pretraining-based model using pretrained contrastive language-vision models and a deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining-based approach is more effective at identifying relevant visual content than directly trained deep autoregressive models.

cs.CV↗

arXiv2024

Integrated adaptive coherent LiDAR for 4D bionic vision

Light detection and ranging (LiDAR) is a ubiquitous tool to provide precise spatial awareness in various perception environments. A bionic LiDAR that can mimic human-like vision by adaptively gazing at selected regions of interest within a broad field of view is crucial to achieve high-resolution imaging in an energy-saving and cost-effective manner. However, current LiDARs based on stacking fixed-wavelength laser arrays and inertial scanning have not been able to achieve the desired dynamic focusing patterns and agile scalability simultaneously. Moreover, the ability to synchronously acquire multi-dimensional physical parameters, including distance, direction, Doppler, and color, through seamless fusion between multiple sensors, still remains elusive in LiDAR. Here, we overcome these limitations and demonstrate a bio-inspired frequency-modulated continuous wave (FMCW) LiDAR system with dynamic and scalable gazing capability. Our chip-scale LiDAR system is built using hybrid integrated photonic solutions, where a frequency-chirped external cavity laser provides broad spectral tunability, while on-chip electro-optic combs with elastic channel spacing allow customizable imaging granularity. Using the dynamic zoom-in capability and the coherent FMCW scheme, we achieve a state-of-the-art resolution of 0.012 degrees, providing up to 15 times the resolution of conventional 3D LiDAR sensors, with 115 equivalent scanning lines and 4D parallel imaging. We further demonstrate cooperative sensing between our adaptive coherent LiDAR and a camera to enable high-resolution color-enhanced machine vision.

physics.optics↗

arXiv2024

A New Dataset, Notation Software, and Representation for Computational Schenkerian Analysis

Schenkerian Analysis (SchA) is a uniquely expressive method of music analysis, combining elements of melody, harmony, counterpoint, and form to describe the hierarchical structure supporting a work of music. However, despite its powerful analytical utility and potential to improve music understanding and generation, SchA has rarely been utilized by the computer music community. This is in large part due to the paucity of available high-quality data in a computer-readable format. With a larger corpus of Schenkerian data, it may be possible to infuse machine learning models with a deeper understanding of musical structure, thus leading to more "human" results. To encourage further research in Schenkerian analysis and its potential benefits for music informatics and generation, this paper presents three main contributions: 1) a new and growing dataset of SchAs, the largest in human- and computer-readable formats to date (>140 excerpts), 2) a novel software for visualization and collection of SchA data, and 3) a novel, flexible representation of SchA as a heterogeneous-edge graph data structure.

cs.SD↗

arXiv2023

Equipping Pretrained Unconditional Music Transformers with Instrument and Genre Controls

The ''pretraining-and-finetuning'' paradigm has become a norm for training domain-specific models in natural language processing and computer vision. In this work, we aim to examine this paradigm for symbolic music generation through leveraging the largest ever symbolic music dataset sourced from the MuseScore forum. We first pretrain a large unconditional transformer model using 1.5 million songs. We then propose a simple technique to equip this pretrained unconditional music transformer model with instrument and genre controls by finetuning the model with additional control tokens. Our proposed representation offers improved high-level controllability and expressiveness against two existing representations. The experimental results show that the proposed model can successfully generate music with user-specified instruments and genre. In a subjective listening test, the proposed model outperforms the pretrained baseline model in terms of coherence, harmony, arrangement and overall quality.

cs.SD↗

arXiv2020

A silicon integrated microwave photonic beamformer

Optical beamforming networks (OBFNs) based on optical true time delay lines (OTTDLs) are well-known as the promising candidate to solve the bandwidth limitation of traditional electronic phased array antennas (PAAs) due to beam squinting. Here we report the first monolithic 1x8 microwave photonic beamformer based on switchable OTTDLs on the silicon-on-insulator platform. The chip consists of a modulator, an eight-channel OBFN, and 8 photodetectors, which includes hundreds of active and passive components in total. It has a wide operating bandwidth from 8 to 18 GHz, which is almost two orders larger than that of electronic PAAs. The beam can be steered to 31 distinguishable angles in the range of -75.51° to 75.64° based on the beam pattern calculation with the measured RF response. The response time for beam steering is 56 μs. These results represent a significant step towards the realization of integrated microwave photonic beamformers that can satisfy compact size and low power consumption requirements for the future radar and wireless communication systems.

physics.optics↗