arXiv subjects

Ao Zhang

Publications and source records attributed to Ao Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2025

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy in the human world. A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapidly learn new tasks. To this end, we introduce GR00T N1, an open foundation model for humanoid robots. GR00T N1 is a Vision-Language-Action (VLA) model with a dual-system architecture. The vision-language module (System 2) interprets the environment through vision and language instructions. The subsequent diffusion transformer module (System 1) generates fluid motor actions in real time. Both modules are tightly coupled and jointly trained end-to-end. We train GR00T N1 with a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets. We show that our generalist robot model GR00T N1 outperforms the state-of-the-art imitation learning baselines on standard simulation benchmarks across multiple robot embodiments. Furthermore, we deploy our model on the Fourier GR-1 humanoid robot for language-conditioned bimanual manipulation tasks, achieving strong performance with high data efficiency.

cs.RO↗

arXiv2025

Random attractors for damped stochastic fractional Schrödinger equation on $\mathbb{R}^{n}$

We study the random attractors associated with the stochastic fractional Schrödinger equation on $\mathbb{R}^n$. Utilizing the stochastic Strichartz estimates for the damped fractional Schrödinger equation with Gaussian noise, we show the existence and uniqueness of a global solution to the damped stochastic fractional nonlinear Schrödinger equation in $H^α(\mathbb{R}^n)$. Furthermore, we demonstrate that this equation defines an infinite-dimensional dynamical system, which possesses a global attractor in $H^α(\mathbb{R}^n)$.

math.AP↗

arXiv2025

Topological Charge Quadrupole Protected by Spin-Orbit U(1) Quasi-Symmetry in Antiferromagnet NdBiPt

The interplay of symmetry and topology in crystal solids has given rise to various elementary excitations as quasiparticles. Among these, those with significant Berry-phase-related transport responses are of particular interest. Here, we predict a new type of quasiparticle called topological charge quadruple (TCQ), which is analogous to a charge quadrupole but consists of two closely-packed pairs of Weyl points in momentum space, specifically in a half-Heusler antiferromagnet NdBiPt. Interestingly, the TCQ is protected by the spin-orbit $U(1)$ quasi-symmetry, rather than any exact crystallographic symmetries. This quasi-symmetry restricts the energy splitting induced by symmetry-lowering perturbations to a second-order effect. Furthermore, the closely located Berry curvature sources and sinks in the TCQ lead to a large Berry curvature dipole, resulting in a significant nonlinear Hall effect. Our work opens an avenue for designing novel quasiparticles using quasi-symmetries and developing materials with enhanced nonlinear responses.

cond-mat.mtrl-sci↗

arXiv2024

Catalog of Unconventional Magnons in Collinear Magnets

Topological magnons have garnered significant interest for their potential in both fundamental research and device applications, owing to their exotic, uncharged, yet topologically protected boundary modes. However, their comprehension has been hindered by the absence of fundamental symmetry descriptions of magnetic materials, which are primarily governed by isotropic Heisenberg interactions in spin Hamiltonians. The ensuing magnon dispersions enable gapless magnon band nodes that go beyond the scenario of representation theory of the magnetic space groups (MSGs), thus referred to as unconventional magnons. Here we developed spin space group (SSG) theory to elucidate collinear magnetic configurations, classifying the 1421 collinear SSGs into four types, constructing their band representations, and providing a comprehensive tabulation of unconventional magnons, such as duodecuple points, octuple nodal lines, and charge-4 octuple points. Based on the MAGNDATA database, we identified 498 collinear magnets with unconventional magnons, among which over 200 magnon band structures were obtained by using first-principles calculations and linear spin wave theory. Additionally, we evaluated the influence of the spin-orbit coupling-induced exchange interaction in these magnets and found that more than 80% are predominantly governed by the Heisenberg interactions, indicating that SSG serves as an ideal framework for describing magnon band nodes in most 3d, 4d and half-filled 4f collinear magnets. Our work offers new pathways for exploring uncharged transports in magnonic systems, holding promise for advancements in next-generation spintronic devices.

cond-mat.mtrl-sci↗

arXiv2024

Weak pullback attractors for damped stochastic fractional Schrödinger equation on $\mathbb{R}^n

This article discusses the weak pullback attractors for a damped stochastic fractional Schrödinger equation on $\mathbb{R}^n$ with $n\geq 2$. By utilizing the stochastic Strichartz estimates and a stopping time technique argument, the existence and uniqueness of a global solution for the systems with the nonlinear term $|u|^{2σ}u$ are proven. Furthermore, we define a mean random dynamical system due to the uniqueness of the solution, which has a unique weak pullback mean random attractor in $L^ρ\left(Ω; L^2\left(\mathbb{R}^n\right)\right)$. This result highlights the long-term dynamics of a broad class of stochastic fractional dispersion equations.

math.AP↗

arXiv2024

Configurable Foundation Models: Building LLMs from a Modular Perspective

Advancements in LLMs have recently unveiled challenges tied to computational efficiency and continual scalability due to their requirements of huge parameters, making the applications and evolution of these models on devices with limited computation resources and scenarios requiring various abilities increasingly cumbersome. Inspired by modularity within the human brain, there is a growing tendency to decompose LLMs into numerous functional modules, allowing for inference with part of modules and dynamic assembly of modules to tackle complex tasks, such as mixture-of-experts. To highlight the inherent efficiency and composability of the modular approach, we coin the term brick to represent each functional module, designating the modularized structure as configurable foundation models. In this paper, we offer a comprehensive overview and investigation of the construction, utilization, and limitation of configurable foundation models. We first formalize modules into emergent bricks - functional neuron partitions that emerge during the pre-training phase, and customized bricks - bricks constructed via additional post-training to improve the capabilities and knowledge of LLMs. Based on diverse functional bricks, we further present four brick-oriented operations: retrieval and routing, merging, updating, and growing. These operations allow for dynamic configuration of LLMs based on instructions to handle complex tasks. To verify our perspective, we conduct an empirical analysis on widely-used LLMs. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions. Finally, we highlight several open issues and directions for future research. Overall, this paper aims to offer a fresh modular perspective on existing LLM research and inspire the future creation of more efficient and scalable foundational models.

cs.AI↗

arXiv2024

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone. However, significant challenges remain preventing MLLMs from being practical in real-world applications. The most notable challenge comes from the huge cost of running an MLLM with a massive number of parameters and extensive computation. As a result, most MLLMs need to be deployed on high-performing cloud servers, which greatly limits their application scopes such as mobile, offline, energy-sensitive, and privacy-protective scenarios. In this work, we present MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By integrating the latest MLLM techniques in architecture, pretraining and alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1) Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong OCR capability and 1.8M pixel high-resolution image perception at any aspect ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual support for 30+ languages, and (5) efficient deployment on mobile phones. More importantly, MiniCPM-V can be viewed as a representative example of a promising trend: The model sizes for achieving usable (e.g., GPT-4V) level performance are rapidly decreasing, along with the fast growth of end-side computation capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices are becoming increasingly possible, unlocking a wider spectrum of real-world AI applications in the near future.

cs.CV↗

arXiv2024

Group Theory on Quasisymmetry and Protected Near Degeneracy

In solid state systems, group representation theory is powerful in characterizing the behavior of quasiparticles, notably the energy degeneracy. While conventional group theory is effective in answering yes-or-no questions related to symmetry breaking, its application to determining the magnitude of energy splitting resulting from symmetry lowering is limited. Here, we propose a theory on quasisymmetry and near degeneracy, thereby expanding the applicability of group theory to address questions regarding large-or-small energy splitting. Defined within the degenerate subspace of an unperturbed Hamiltonian, quasisymmetries form an enlarged symmetry group eliminating the first-order splitting. This framework ensures that the magnitude of splitting arises as a second-order effect of symmetry-lowering perturbations, such as external fields and spin-orbit coupling. We systematically tabulate the quasisymmetry groups within 32 crystallographic point groups and find all the possible unitary quasisymmetry group structures regarding double degeneracy. Applying our theory to the realistic material AgLa, we predict a "quasi-Dirac semimetal" phase characterized by two tiny-gap band anticrossings.

cond-mat.mtrl-sci↗

arXiv2024

Enumeration and representation theory of spin space groups

Those fundamental physical properties, such as phase transitions, Weyl fermions, and spin excitation, in all magnetic ordered materials, were ultimately believed to rely on the symmetry theory of magnetic space groups. Recently, it has come to light that a more comprehensive group, known as the spin space group (SSG), which combines separate spin and spatial operations, is necessary to fully characterize the geometry and underlying properties of magnetic ordered materials. However, the basic theory of SSG has seldom been developed. In this work, we present a systematic study of the enumeration and the representation theory of SSG. Starting from the 230 crystallographic space groups and finite translation groups with a maximum order of 8, we establish an extensive collection of over 100000 SSGs under a four-index nomenclature as well as the International notation. We then identify inequivalent SSGs specifically applicable to collinear, coplanar, and noncoplanar magnetic configurations. To facilitate the identification of SSG, we develop an online program (findspingroup.com) that can determine the SSG symmetries of any magnetic ordered crystals. Moreover, we derive the irreducible co-representations of the little group in momentum space within the SSG framework. Finally, we illustrate the SSG symmetries and physical effects beyond the framework of magnetic space groups through several representative material examples, including a well-known altermagnet RuO2, spiral spin polarization in the coplanar antiferromagnet CeAuAl3, and geometric Hall effect in the noncoplanar antiferromagnet CoNb3S6. Our work advances the field of group theory in describing magnetic ordered materials, opening up avenues for deeper comprehension and further exploration of emergent phenomena in magnetic materials.

cond-mat.mtrl-sci↗

arXiv2024

The stochastic fractional nonlinear Schrödinger equations in $H^α$ and structure-preserving algorithm

In this paper, we first investigate the global existence of a solution for the stochastic fractional nonlinear Schrödinger equation with radially symmetric initial data in a suitable energy space $H^α$. We then show that the stochastic fractional nonlinear Schrödinger equation in the Stratonovich sense forms an infinite-dimensional stochastic Hamiltonian system, with its phase flow preserving symplecticity. Finally, we develop a stochastic midpoint scheme for the stochastic fractional nonlinear Schrödinger equation from the perspective of symplectic geometry. It is proved that the stochastic midpoint scheme satisfies the corresponding symplectic law in the discrete sense. A numerical example is conducted to validate the efficiency of the theory.

math.NA↗

arXiv2024

Physical formula enhanced multi-task learning for pharmacokinetics prediction

Artificial intelligence (AI) technology has demonstrated remarkable potential in drug dis-covery, where pharmacokinetics plays a crucial role in determining the dosage, safety, and efficacy of new drugs. A major challenge for AI-driven drug discovery (AIDD) is the scarcity of high-quality data, which often requires extensive wet-lab work. A typical example of this is pharmacokinetic experiments. In this work, we develop a physical formula enhanced mul-ti-task learning (PEMAL) method that predicts four key parameters of pharmacokinetics simultaneously. By incorporating physical formulas into the multi-task framework, PEMAL facilitates effective knowledge sharing and target alignment among the pharmacokinetic parameters, thereby enhancing the accuracy of prediction. Our experiments reveal that PEMAL significantly lowers the data demand, compared to typical Graph Neural Networks. Moreover, we demonstrate that PEMAL enhances the robustness to noise, an advantage that conventional Neural Networks do not possess. Another advantage of PEMAL is its high flexibility, which can be potentially applied to other multi-task machine learning scenarios. Overall, our work illustrates the benefits and potential of using PEMAL in AIDD and other scenarios with data scarcity and noise.

q-bio.QM↗

arXiv2024

ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge

To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge. This challenge collects over 100 hours of multi-channel speech data recorded inside a new energy vehicle and 40 hours of noise for data augmentation. Two tracks, including automatic speech recognition (ASR) and automatic speech diarization and recognition (ASDR) are set up, using character error rate (CER) and concatenated minimum permutation character error rate (cpCER) as evaluation metrics, respectively. Overall, the ICMC-ASR Challenge attracts 98 participating teams and receives 53 valid results in both tracks. In the end, first-place team USTCiflytek achieves a CER of 13.16% in the ASR track and a cpCER of 21.48% in the ASDR track, showing an absolute improvement of 13.08% and 51.4% compared to our challenge baseline, respectively.

cs.SD↗

arXiv2023

NExT-Chat: An LMM for Chat, Detection and Segmentation

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat.

cs.CV↗

arXiv2023

Chiral Dirac fermion in a collinear antiferromagnet

In a Dirac semimetal, the massless Dirac fermion has zero chirality, leading to surface states connected adiabatically to a topologically trivial surface state as well as vanishing anomalous Hall effect (AHE). Recently, it is predicted that in the nonrelativistic limit of certain collinear antiferromagnets, there exists a type of chiral Dirac-like fermion, whose dispersion manifests four-fold degenerate crossing points formed by spin-degenerate linear bands, with topologically protected Fermi arcs. Such unconventional chiral fermion, protected by a hidden SU(2) symmetry in the hierarchy of an enhanced crystallographic group, namely spin space group, is not experimentally verified yet. Here, by angle-resolved photoemission spectroscopy measurements, we reveal the surface origin of the electron pocket at the Fermi surface in collinear antiferromagnet CoNb3S6. Combining with neutron diffraction and first-principles calculations, we suggest a multidomain collinear AFM configuration, rendering the the existence of the Fermi-arc surface states induced by chiral Dirac-like fermions. Our work provides spectral evidence of the chiral Dirac-like fermion caused by particular spin symmetry in CoNb3S6, paving an avenue for exploring new emergent phenomena in antiferromagnets with unconventional quasiparticle excitations.

cond-mat.mtrl-sci↗

arXiv2023

U2-KWS: Unified Two-pass Open-vocabulary Keyword Spotting with Keyword Bias

Open-vocabulary keyword spotting (KWS), which allows users to customize keywords, has attracted increasingly more interest. However, existing methods based on acoustic models and post-processing train the acoustic model with ASR training criteria to model all phonemes, making the acoustic model under-optimized for the KWS task. To solve this problem, we propose a novel unified two-pass open-vocabulary KWS (U2-KWS) framework inspired by the two-pass ASR model U2. Specifically, we employ the CTC branch as the first stage model to detect potential keyword candidates and the decoder branch as the second stage model to validate candidates. In order to enhance any customized keywords, we redesign the U2 training procedure for U2-KWS and add keyword information by audio and text cross-attention into both branches. We perform experiments on our internal dataset and Aishell-1. The results show that U2-KWS can achieve a significant relative wake-up rate improvement of 41% compared to the traditional customized KWS systems when the false alarm rate is fixed to 0.5 times per hour.

eess.AS↗

arXiv2023

VPGTrans: Transfer Visual Prompt Generator across LLMs

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the MLLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing MLLMs for the target MLLM. In this work, we for the first time investigate the VPG transferability across LLMs, and explore a solution to reduce the cost of VPG transfer. We first study the VPG transfer across different LLM sizes (e.g., small-to-large), and across different LLM types, through which we diagnose the key factors to maximize the transfer efficiency. Based on our observation, we design a two-stage transfer framework named VPGTrans, which is simple yet highly effective. Through extensive experiments, we demonstrate that VPGTrans helps significantly speed up the transfer learning process without compromising performance. Remarkably, it helps achieve the VPG transfer from BLIP-2 OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and 10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from scratch. Further, a series of intriguing findings and potential rationales behind them are provided and discussed. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

cs.CV↗

arXiv2023

Spike-Triggered Contextual Biasing for End-to-End Mandarin Speech Recognition

The attention-based deep contextual biasing method has been demonstrated to effectively improve the recognition performance of end-to-end automatic speech recognition (ASR) systems on given contextual phrases. However, unlike shallow fusion methods that directly bias the posterior of the ASR model, deep biasing methods implicitly integrate contextual information, making it challenging to control the degree of bias. In this study, we introduce a spike-triggered deep biasing method that simultaneously supports both explicit and implicit bias. Moreover, both bias approaches exhibit significant improvements and can be cascaded with shallow fusion methods for better results. Furthermore, we propose a context sampling enhancement strategy and improve the contextual phrase filtering algorithm. Experiments on the public WenetSpeech Mandarin biased-word dataset show a 32.0% relative CER reduction compared to the baseline model, with an impressively 68.6% relative CER reduction on contextual phrases.

eess.AS↗

arXiv2023

Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition

By incorporating additional contextual information, deep biasing methods have emerged as a promising solution for speech recognition of personalized words. However, for real-world voice assistants, always biasing on such personalized words with high prediction scores can significantly degrade the performance of recognizing common words. To address this issue, we propose an adaptive contextual biasing method based on Context-Aware Transformer Transducer (CATT) that utilizes the biased encoder and predictor embeddings to perform streaming prediction of contextual phrase occurrences. Such prediction is then used to dynamically switch the bias list on and off, enabling the model to adapt to both personalized and common scenarios. Experiments on Librispeech and internal voice assistant datasets show that our approach can achieve up to 6.7% and 20.7% relative reduction in WER and CER compared to the baseline respectively, mitigating up to 96.7% and 84.9% of the relative WER and CER increase for common cases. Furthermore, our approach has a minimal performance impact in personalized scenarios while maintaining a streaming inference pipeline with negligible RTF increase.

cs.SD↗