Search arXiv⌕ Search

arXiv subjects

Xu He

Publications and source records attributed to Xu He.

At least 73 records · Page 4Linked to original sources
arXiv2025

Landmarks Are Alike Yet Distinct: Harnessing Similarity and Individuality for One-Shot Medical Landmark Detection

Landmark detection plays a crucial role in medical imaging applications such as disease diagnosis, bone age estimation, and therapy planning. However, training models for detecting multiple landmarks simultaneously often encounters the "seesaw phenomenon", where improvements in detecting certain landmarks lead to declines in detecting others. Yet, training a separate model for each landmark increases memory usage and computational overhead. To address these challenges, we propose a novel approach based on the belief that "landmarks are distinct" by training models with pseudo-labels and template data updated continuously during the training process, where each model is dedicated to detecting a single landmark to achieve high accuracy. Furthermore, grounded on the belief that "landmarks are also alike", we introduce an adapter-based fusion model, combining shared weights with landmark-specific weights, to efficiently share model parameters while allowing flexible adaptation to individual landmarks. This approach not only significantly reduces memory and computational resource requirements but also effectively mitigates the seesaw phenomenon in multi-landmark training. Experimental results on publicly available medical image datasets demonstrate that the single-landmark models significantly outperform traditional multi-point joint training models in detecting individual landmarks. Although our adapter-based fusion model shows slightly lower performance compared to the combined results of all single-landmark models, it still surpasses the current state-of-the-art methods while achieving a notable improvement in resource efficiency.

cs.CV↗
arXiv2025

Visible-Thermal Tiny Object Detection: A Benchmark Dataset and Baselines

Small object detection (SOD) has been a longstanding yet challenging task for decades, with numerous datasets and algorithms being developed. However, they mainly focus on either visible or thermal modality, while visible-thermal (RGBT) bimodality is rarely explored. Although some RGBT datasets have been developed recently, the insufficient quantity, limited category, misaligned images and large target size cannot provide an impartial benchmark to evaluate multi-category visible-thermal small object detection (RGBT SOD) algorithms. In this paper, we build the first large-scale benchmark with high diversity for RGBT SOD (namely RGBT-Tiny), including 115 paired sequences, 93K frames and 1.2M manual annotations. RGBT-Tiny contains abundant targets (7 categories) and high-diversity scenes (8 types that cover different illumination and density variations). Note that, over 81% of targets are smaller than 16x16, and we provide paired bounding box annotations with tracking ID to offer an extremely challenging benchmark with wide-range applications, such as RGBT fusion, detection and tracking. In addition, we propose a scale adaptive fitness (SAFit) measure that exhibits high robustness on both small and large targets. The proposed SAFit can provide reasonable performance evaluation and promote detection performance. Based on the proposed RGBT-Tiny dataset and SAFit measure, extensive evaluations have been conducted, including 23 recent state-of-the-art algorithms that cover four different types (i.e., visible generic detection, visible SOD, thermal SOD and RGBT object detection). Project is available at https://github.com/XinyiYing/RGBT-Tiny.

cs.CV↗
arXiv2025

TYPEPULSE: Detecting Type Confusion Bugs in Rust Programs

Rust supports type conversions and safe Rust guarantees the security of these conversions through robust static type checking and strict ownership guidelines. However, there are instances where programmers need to use unsafe Rust for certain type conversions, especially those involving pointers. Consequently, these conversions may cause severe memory corruption problems. Despite extensive research on type confusion bugs in C/C++, studies on type confusion bugs in Rust are still lacking. Also, due to Rust's new features in the type system, existing solutions in C/C++ cannot be directly applied to Rust. In this paper, we develop a static analysis tool called TYPEPULSE to detect three main categories of type confusion bugs in Rust including misalignment, inconsistent layout, and mismatched scope. TYPEPULSE first performs a type conversion analysis to collect and determine trait bounds for type pairs. Moreover, it performs a pointer alias analysis to resolve the alias relationship of pointers. Following the integration of information into the property graph, it constructs type patterns and detects each type of bug in various conversion scenarios. We run TYPEPULSE on the top 3,000 Rust packages and uncover 71 new type confusion bugs, exceeding the total number of type confusion bugs reported in RUSTSEC over the past five years. We have received 32 confirmations from developers, along with one CVE ID and six RUSTSEC IDs.

cs.CR↗
arXiv2025

Stacking-engineered ferroelectricity and multiferroic order in van der Waals magnets

Two-dimensional (2D) materials that exhibit spontaneous magnetization, polarization or strain (referred to as ferroics) have the potential to revolutionize nanotechnology by enhancing the multifunctionality of nanoscale devices. However, multiferroic order is difficult to achieve, requiring complicated coupling between electron and spin degrees of freedom. We propose a universal method to engineer multiferroics from van der Waals magnets by taking advantage of the fact that changing the stacking between 2D layers can break inversion symmetry, resulting in ferroelectricity and possibly magnetoelectric coupling. We illustrate this concept using first-principles calculations in bilayer NiI$_2$, which can be made ferroelectric upon rotating two adjacent layers by $180^{\circ}$ with respect to the bulk stacking. Furthermore, we discover a novel multiferroic order induced by interlayer charge transfer which couples the interlayer spin order and electronic polarization. Our approach is not only general but also systematic, and can enable the discovery of a wide variety of 2D multiferroics.

cond-mat.mtrl-sci↗
arXiv2025

Orbital Order Triggered Out-of-Plane Ferroelectricity in Magnetic Transition Metal di-halide Monolayers

Although multiferroics have undergone extensive examination for several decades, the occurrence of ferroelectricity induced by orbital order is only scarcely documented. In this study, we propose the existence of spontaneous ferroelectric states featuring a finite out-of-plane polarization in monolayer compounds of magnetic transition metal di-halides. Our first principles analysis reveals that partially occupied d-orbital states within octahedra exhibit a preference for spatial orbital order within a two-dimensional lattice. The absence of inversion symmetry, arising from orbital order, serves as the driving force introducing additional electric polarization along the out-of-plane direction. Unlike previous reported orbital orders arising from metal states in lattice, the non-colinear ones we studied in this work relate to the transition between two insulator states. The resultant asymmetric Jahn-Teller distortions are accompanied as the consequence producing additional ionic polarization. Importantly, our findings indicate that this mechanism is not confined to a specific material but is a shared characteristic among a series of monolayer transition metal magnetic di-halides, proposing an innovative form of intrinsic two-dimensional multiferroic physics.

cond-mat.mtrl-sci↗
arXiv2024

The anti-distortive polaron : an alternative mechanism for lattice-mediated charge trapping

Polarons can naturally form in materials from the interaction of extra charge carriers with the atomic lattice. Ubiquitous, they are central to various topics and phenomena such as high-T$_c$ superconductivity, electrochromism, photovoltaics, photocatalysis or ion batteries. However, polaron formation remains poorly understood and mostly relies on few historical models such as Landau-Pekar, Frölich, Holstein or Jahn-Teller polarons. Here, from advanced first-principles calculations, we show that the formation of intriguing medium-size polarons in WO$_3$ does not fit with traditional models but instead arises from the undoing of distortive atomic motions inherent to the pristine phase, which lowers the bandgap through dynamical covalency effects. We so introduce the innovative concept of {\it anti-distortive} polaron and rationalize it from a quantum-dot model. We demonstrate that anti-distortive polarons are generic to different families of compounds and clarify how this new concept opens concrete perspectives for a better control of the polaronic state and related properties.

cond-mat.mtrl-sci↗
arXiv2024

iVideoGPT: Interactive VideoGPTs are Scalable World Models

World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals--visual observations, actions, and rewards--into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

cs.CV↗
arXiv2024

Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection

Unsupervised out-of-distribution (U-OOD) detection is to identify OOD data samples with a detector trained solely on unlabeled in-distribution (ID) data. The likelihood function estimated by a deep generative model (DGM) could be a natural detector, but its performance is limited in some popular "hard" benchmarks, such as FashionMNIST (ID) vs. MNIST (OOD). Recent studies have developed various detectors based on DGMs to move beyond likelihood. However, despite their success on "hard" benchmarks, most of them struggle to consistently surpass or match the performance of likelihood on some "non-hard" cases, such as SVHN (ID) vs. CIFAR10 (OOD) where likelihood could be a nearly perfect detector. Therefore, we appeal for more attention to incremental effectiveness on likelihood, i.e., whether a method could always surpass or at least match the performance of likelihood in U-OOD detection. We first investigate the likelihood of variational DGMs and find its detection performance could be improved in two directions: i) alleviating latent distribution mismatch, and ii) calibrating the dataset entropy-mutual integration. Then, we apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. The final method, named Resultant, combines these two directions for better incremental effectiveness compared to either technique alone. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector while maintaining incremental effectiveness on likelihood in a wide range of tasks.

stat.ML↗
arXiv2024

MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement

Existing works in single-image human reconstruction suffer from weak generalizability due to insufficient training data or 3D inconsistencies for a lack of comprehensive multi-view knowledge. In this paper, we introduce MagicMan, a human-specific multi-view diffusion model designed to generate high-quality novel view images from a single reference image. As its core, we leverage a pre-trained 2D diffusion model as the generative prior for generalizability, with the parametric SMPL-X model as the 3D body prior to promote 3D awareness. To tackle the critical challenge of maintaining consistency while achieving dense multi-view generation for improved 3D human reconstruction, we first introduce hybrid multi-view attention to facilitate both efficient and thorough information interchange across different views. Additionally, we present a geometry-aware dual branch to perform concurrent generation in both RGB and normal domains, further enhancing consistency via geometry cues. Last but not least, to address ill-shaped issues arising from inaccurate SMPL-X estimation that conflicts with the reference image, we propose a novel iterative refinement strategy, which progressively optimizes SMPL-X accuracy while enhancing the quality and consistency of the generated multi-views. Extensive experimental results demonstrate that our method significantly outperforms existing approaches in both novel view synthesis and subsequent 3D human reconstruction tasks.

cs.CV↗
arXiv2024

Finite-temperature properties of antiferroelectric perovskite $\rm PbZrO_3$ from deep learning interatomic potential

The prototypical antiferroelectric perovskite $\rm PbZrO_3$ (PZO) has garnered considerable attentions in recent years due to its significance in technological applications and fundamental research. Many unresolved issues in PZO are associated with large length- and time-scales, as well as finite temperatures, presenting significant challenges for first-principles density functional theory studies. Here, we introduce a deep learning interatomic potential of PZO, enabling investigation of finite-temperature properties through large-scale atomistic simulations. Trained using an elaborately designed dataset, the model successfully reproduces a large number of phases, in particular, the recently discovered 80-atom antiferroelectric $Pnam$ phase and ferrielectric $Ima2$ phase, providing precise predictions for their structural and dynamical properties. Using this model, we investigated phase transitions of multiple phases, including $Pbam$/$Pnam$, $Ima2$ and $R3c$, which show high similarity to the experimental observation. Our simulation results also highlight the crucial role of free-energy in determining the low-temperature phase of PZO, reconciling the apparent contradiction: $Pbam$ is the most commonly observed phase in experiments, while theoretical calculations predict other phases exhibiting even lower energy. Furthermore, in the temperature range where the $Pbam$ phase is thermodynamically stable, typical double polarization hysteresis loops for antiferroelectrics were obtained, along with a detailed elucidation of the structural evolution during the electric-field induced transitions between the non-polar $Pbam$ and polar $R3c$ phases.

cond-mat.mtrl-sci↗
arXiv2024

MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time

Although Large Language Models (LLMs) achieve remarkable performance across various tasks, they often struggle with complex reasoning tasks, such as answering mathematical questions. Recent efforts to address this issue have primarily focused on leveraging mathematical datasets through supervised fine-tuning or self-improvement techniques. However, these methods often depend on high-quality datasets that are difficult to prepare, or they require substantial computational resources for fine-tuning. Inspired by findings that LLMs know how to produce the right answer but struggle to select the correct reasoning path, we propose a purely inference-based searching method -- MindStar (M*). This method formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. We evaluate the M* framework on both the GSM8K and MATH datasets, comparing its performance with existing open and closed-source LLMs. Our results demonstrate that M* significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1, but with substantially reduced model size and computational costs.

cs.LG↗
arXiv2024

Tunable Collective Excitations in Epitaxial Perovskite Nickelates

The formation of plasmons through the collective excitation of charge density has generated intense discussions, offering insights to fundamental sciences and potential applications. While the underlying physical principles have been well-established, the effects of many-body interactions and orbital hybridization on plasmonic dynamics remain understudied. In this work, we present the observation of conventional metallic and correlated plasmons in epitaxial La1-xSrxNiO3 (LSNO) films with varying Sr doping concentrations (x = 0, 0.125, 0.25), unveiling their intriguing evolution. Unlike samples at other doping concentrations, the x = 0.125 intermediate doping sample does not exhibit the correlated plasmons despite showing high optical conductivity. Through a comprehensive experimental investigation using spectroscopic ellipsometry and X-ray absorption spectroscopy, the O2p-Ni3d orbital hybridization for LSNO with a doping concentration of x = 0.125 is found to be significantly enhanced, alongside a considerable weakening of its effective correlation U*. These factors account for the absence of correlated plasmons and the high optical conductivity observed in LSNO (0.125). Our results underscore the profound impact of orbital hybridization on the electronic structure and the formation of plasmon in strongly-correlated systems. This in turn suggest that LSNO could serve as a promising alternative material in optoelectronic devices.

cond-mat.mtrl-sci↗
arXiv2024

Uncovering an Interfacial Band Resulting from Orbital Hybridization in Nickelate Heterostructures

The interaction of atomic orbitals at the interface of perovskite oxide heterostructures has been investigated for its profound impact on the band structures and electronic properties, giving rise to unique electronic states and a variety of tunable functionalities. In this study, we conducted an extensive investigation of the optical and electronic properties of epitaxial NdNiO3 thin films grown on a series of single crystal substrates. Unlike films synthesized on other substrates, NdNiO3 on SrTiO3 (NNO/STO) gives rise to a unique band structure which features an additional unoccupied band situated above the Fermi level. Our comprehensive investigation, which incorporated a wide array of experimental techniques and density functional theory calculations, revealed that the emergence of the interfacial band structure is primarily driven by the orbital hybridization between Ti 3d orbitals of the STO substrate and O 2p orbitals of the NNO thin film. Furthermore, exciton peaks have been detected in the optical spectra of the NNO/STO film, attributable to the pronounced electron-electron (e-e) and electron-hole (e-h) interactions propagating from the STO substrate into the NNO film. These findings underscore the substantial influence of interfacial orbital hybridization on the electronic structure of oxide thin-films, thereby offering key insights into tuning their interfacial properties.

cond-mat.mtrl-sci↗
arXiv2024

Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model

Co-speech gestures, if presented in the lively form of videos, can achieve superior visual effects in human-machine interaction. While previous works mostly generate structural human skeletons, resulting in the omission of appearance information, we focus on the direct generation of audio-driven co-speech gesture videos in this work. There are two main challenges: 1) A suitable motion feature is needed to describe complex human movements with crucial appearance information. 2) Gestures and speech exhibit inherent dependencies and should be temporally aligned even of arbitrary length. To solve these problems, we present a novel motion-decoupled framework to generate co-speech gesture videos. Specifically, we first introduce a well-designed nonlinear TPS transformation to obtain latent motion features preserving essential appearance information. Then a transformer-based diffusion model is proposed to learn the temporal correlation between gestures and speech, and performs generation in the latent motion space, followed by an optimal motion selection module to produce long-term coherent and consistent gesture videos. For better visual perception, we further design a refinement network focusing on missing details of certain areas. Extensive experimental results show that our proposed framework significantly outperforms existing approaches in both motion and video-related evaluations. Our code, demos, and more resources are available at https://github.com/thuhcsi/S2G-MDDiffusion.

cs.CV↗
arXiv2024

Enhancing Expressiveness in Dance Generation via Integrating Frequency and Music Style Information

Dance generation, as a branch of human motion generation, has attracted increasing attention. Recently, a few works attempt to enhance dance expressiveness, which includes genre matching, beat alignment, and dance dynamics, from certain aspects. However, the enhancement is quite limited as they lack comprehensive consideration of the aforementioned three factors. In this paper, we propose ExpressiveBailando, a novel dance generation method designed to generate expressive dances, concurrently taking all three factors into account. Specifically, we mitigate the issue of speed homogenization by incorporating frequency information into VQ-VAE, thus improving dance dynamics. Additionally, we integrate music style information by extracting genre- and beat-related features with a pre-trained music model, hence achieving improvements in the other two factors. Extensive experimental results demonstrate that our proposed method can generate dances with high expressiveness and outperforms existing methods both qualitatively and quantitatively.

cs.MM↗
arXiv2023

BinGo: Identifying Security Patches in Binary Code with Graph Representation Learning

A timely software update is vital to combat the increasing security vulnerabilities. However, some software vendors may secretly patch their vulnerabilities without creating CVE entries or even describing the security issue in their change log. Thus, it is critical to identify these hidden security patches and defeat potential N-day attacks. Researchers have employed various machine learning techniques to identify security patches in open-source software, leveraging the syntax and semantic features of the software changes and commit messages. However, all these solutions cannot be directly applied to the binary code, whose instructions and program flow may dramatically vary due to different compilation configurations. In this paper, we propose BinGo, a new security patch detection system for binary code. The main idea is to present the binary code as code property graphs to enable a comprehensive understanding of program flow and perform a language model over each basic block of binary code to catch the instruction semantics. BinGo consists of four phases, namely, patch data pre-processing, graph extraction, embedding generation, and graph representation learning. Due to the lack of an existing binary security patch dataset, we construct such a dataset by compiling the pre-patch and post-patch source code of the Linux kernel. Our experimental results show BinGo can achieve up to 80.77% accuracy in identifying security patches between two neighboring versions of binary code. Moreover, BinGo can effectively reduce the false positives and false negatives caused by the different compilers and optimization levels.

cs.CR↗
arXiv2023

Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System

With the continuous increase of users and items, conventional recommender systems trained on static datasets can hardly adapt to changing environments. The high-throughput data requires the model to be updated in a timely manner for capturing the user interest dynamics, which leads to the emergence of streaming recommender systems. Due to the prevalence of deep learning-based recommender systems, the embedding layer is widely adopted to represent the characteristics of users, items, and other features in low-dimensional vectors. However, it has been proved that setting an identical and static embedding size is sub-optimal in terms of recommendation performance and memory cost, especially for streaming recommendations. To tackle this problem, we first rethink the streaming model update process and model the dynamic embedding size search as a bandit problem. Then, we analyze and quantify the factors that influence the optimal embedding sizes from the statistics perspective. Based on this, we propose the \textbf{D}ynamic \textbf{E}mbedding \textbf{S}ize \textbf{S}earch (\textbf{DESS}) method to minimize the embedding size selection regret on both user and item sides in a non-stationary manner. Theoretically, we obtain a sublinear regret upper bound superior to previous methods. Empirical results across two recommendation tasks on four public datasets also demonstrate that our approach can achieve better streaming recommendation performance with lower memory cost and higher time efficiency.

cs.IR↗
arXiv2023

MOEF: Modeling Occasion Evolution in Frequency Domain for Promotion-Aware Click-Through Rate Prediction

Promotions are becoming more important and prevalent in e-commerce to attract customers and boost sales, leading to frequent changes of occasions, which drives users to behave differently. In such situations, most existing Click-Through Rate (CTR) models can't generalize well to online serving due to distribution uncertainty of the upcoming occasion. In this paper, we propose a novel CTR model named MOEF for recommendations under frequent changes of occasions. Firstly, we design a time series that consists of occasion signals generated from the online business scenario. Since occasion signals are more discriminative in the frequency domain, we apply Fourier Transformation to sliding time windows upon the time series, obtaining a sequence of frequency spectrum which is then processed by Occasion Evolution Layer (OEL). In this way, a high-order occasion representation can be learned to handle the online distribution uncertainty. Moreover, we adopt multiple experts to learn feature representations from multiple aspects, which are guided by the occasion representation via an attention mechanism. Accordingly, a mixture of feature representations is obtained adaptively for different occasions to predict the final CTR. Experimental results on real-world datasets validate the superiority of MOEF and online A/B tests also show MOEF outperforms representative CTR models significantly.

cs.LG↗