arXiv subjects

Yu Wang

Publications and source records attributed to Yu Wang.

At least 793 records · Page 44Linked to original sources

arXiv2024

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

Diffusion models have achieved significant visual generation quality. However, their significant computational and memory costs pose challenge for their application on resource-constrained mobile devices or even desktop GPUs. Recent few-step diffusion models reduces the inference time by reducing the denoising steps. However, their memory consumptions are still excessive. The Post Training Quantization (PTQ) replaces high bit-width FP representation with low-bit integer values (INT4/8) , which is an effective and efficient technique to reduce the memory cost. However, when applying to few-step diffusion models, existing quantization methods face challenges in preserving both the image quality and text alignment. To address this issue, we propose an mixed-precision quantization framework - MixDQ. Firstly, We design specialized BOS-aware quantization method for highly sensitive text embedding quantization. Then, we conduct metric-decoupled sensitivity analysis to measure the sensitivity of each layer. Finally, we develop an integer-programming-based method to conduct bit-width allocation. While existing quantization methods fall short at W8A8, MixDQ could achieve W8A8 without performance loss, and W4A8 with negligible visual degradation. Compared with FP16, we achieve 3-4x reduction in model size and memory cost, and 1.45x latency speedup.

cs.CV↗

arXiv2024

Position: Towards Implicit Prompt For Text-To-Image Models

Recent text-to-image (T2I) models have had great success, and many benchmarks have been proposed to evaluate their performance and safety. However, they only consider explicit prompts while neglecting implicit prompts (hint at a target without explicitly mentioning it). These prompts may get rid of safety constraints and pose potential threats to the applications of these models. This position paper highlights the current state of T2I models toward implicit prompts. We present a benchmark named ImplicitBench and conduct an investigation on the performance and impacts of implicit prompts with popular T2I models. Specifically, we design and collect more than 2,000 implicit prompts of three aspects: General Symbols, Celebrity Privacy, and Not-Safe-For-Work (NSFW) Issues, and evaluate six well-known T2I models' capabilities under these implicit prompts. Experiment results show that (1) T2I models are able to accurately create various target symbols indicated by implicit prompts; (2) Implicit prompts bring potential risks of privacy leakage for T2I models. (3) Constraints of NSFW in most of the evaluated T2I models can be bypassed with implicit prompts. We call for increased attention to the potential and risks of implicit prompts in the T2I community and further investigation into the capabilities and impacts of implicit prompts, advocating for a balanced approach that harnesses their benefits while mitigating their risks.

cs.CY↗

arXiv2024

Augmenting Textual Generation via Topology Aware Retrieval

Despite the impressive advancements of Large Language Models (LLMs) in generating text, they are often limited by the knowledge contained in the input and prone to producing inaccurate or hallucinated content. To tackle these issues, Retrieval-augmented Generation (RAG) is employed as an effective strategy to enhance the available knowledge base and anchor the responses in reality by pulling additional texts from external databases. In real-world applications, texts are often linked through entities within a graph, such as citations in academic papers or comments in social networks. This paper exploits these topological relationships to guide the retrieval process in RAG. Specifically, we explore two kinds of topological connections: proximity-based, focusing on closely connected nodes, and role-based, which looks at nodes sharing similar subgraph structures. Our empirical research confirms their relevance to text relationships, leading us to develop a Topology-aware Retrieval-augmented Generation framework. This framework includes a retrieval module that selects texts based on their topological relationships and an aggregation module that integrates these texts into prompts to stimulate LLMs for text generation. We have curated established text-attributed networks and conducted comprehensive experiments to validate the effectiveness of this framework, demonstrating its potential to enhance RAG with topological awareness.

cs.IR↗

arXiv2024

MEMORYLLM: Towards Self-Updatable Large Language Models

Existing Large Language Models (LLMs) usually remain static after deployment, which might make it hard to inject new knowledge into the model. We aim to build models containing a considerable portion of self-updatable parameters, enabling the model to integrate new knowledge effectively and efficiently. To this end, we introduce MEMORYLLM, a model that comprises a transformer and a fixed-size memory pool within the latent space of the transformer. MEMORYLLM can self-update with text knowledge and memorize the knowledge injected earlier. Our evaluations demonstrate the ability of MEMORYLLM to effectively incorporate new knowledge, as evidenced by its performance on model editing benchmarks. Meanwhile, the model exhibits long-term information retention capacity, which is validated through our custom-designed evaluations and long-context benchmarks. MEMORYLLM also shows operational integrity without any sign of performance degradation even after nearly a million memory updates. Our code and model are open-sourced at https://github.com/wangyu-ustc/MemoryLLM.

cs.CL↗

arXiv2024

EVI-SAM: Robust, Real-time, Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping

Event cameras are bio-inspired, motion-activated sensors that demonstrate substantial potential in handling challenging situations, such as motion blur and high-dynamic range. In this paper, we proposed EVI-SAM to tackle the problem of 6 DoF pose tracking and 3D reconstruction using monocular event camera. A novel event-based hybrid tracking framework is designed to estimate the pose, leveraging the robustness of feature matching and the precision of direct alignment. Specifically, we develop an event-based 2D-2D alignment to construct the photometric constraint, and tightly integrate it with the event-based reprojection constraint. The mapping module recovers the dense and colorful depth of the scene through the image-guided event-based mapping method. Subsequently, the appearance, texture, and surface mesh of the 3D scene can be reconstructed by fusing the dense depth map from multiple viewpoints using truncated signed distance function (TSDF) fusion. To the best of our knowledge, this is the first non-learning work to realize event-based dense mapping. Numerical evaluations are performed on both publicly available and self-collected datasets, which qualitatively and quantitatively demonstrate the superior performance of our method. Our EVI-SAM effectively balances accuracy and robustness while maintaining computational efficiency, showcasing superior pose tracking and dense mapping performance in challenging scenarios. Video Demo: https://youtu.be/Nn40U4e5Si8.

cs.CV↗

arXiv2024

Homopolar Chemical Bonds Induce In-Plane Anisotropy in Layered Semiconductors

Main-group layered binary semiconductors, in particular, the III-VI alloys in the binary Ga-Te system are attracting increasing interest for a range of practical applications. The III-VI semiconductor, monoclinic gallium monotelluride (m-GaTe), has been recently used in high-sensitivity photodetectors/phototransistors and electronic memory applications due to its anisotropic properties yielding superior optical and electrical performance. Despite these applications, the origin of such anisotropy, namely the complex structural and bonding environments in GaTe nanostructures remain to be fully understood. In the present work, we report a comprehensive atomic-scale characterization of m-GaTe by state-of-the-art element-resolved atomic-scale microscopy experiments, enabling a direct measure of the in-plane anisotropy at the sub-Angstrom level. We show that these experimental images compare well with the results of first-principles modeling. Quantum-chemical bonding analyses provide a detailed picture of the atomic neighbor interactions within the layers, revealing that vertical Ga-Ga homopolar bonds get stronger when they are distorted and rotated, inducing the strong in-plane anisotropy. Beyond GaTe, using a systematic screening over the Materials Project database, we identify four additional low-symmetric layered crystals with similar distorted tetrahedral patterns, namely, GeP, GeAs, SiP, and SiAs. We thereby confirm the homopolar-bond-induced anisotropy is a more generic feature in these layered van-der-Waals materials.

cond-mat.mtrl-sci↗

arXiv2024

Testing the consistency of early and late cosmological parameters with BAO and CMB data

The recent local measurements of the Hubble constant $H_0$, indicate a significant discrepancy of over 5$σ$ compared to the value inferred from \textit{Planck} observations of the cosmic microwave background (CMB). In this paper, we try to understand the origin of this tension by testing the consistency of early and late cosmological parameters in the same observed data. In practice, we simultaneously derive the early and late parameters using baryon acoustic oscillation (BAO) measurements, which provide both low and high-redshift information. To resolve parameter degeneracy, the complementary data from CMB observations are included in the analysis. By using the parameter $ω_m = Ω_mh^2$, we introduce ${\rm ratio}(ω_m)$, defined as the ratio of $ω_m$ which are constrained from high and low-redshift measurements respectively, to quantify the consistency between early and late parameters. We obtained a value of ${\rm ratio}(ω_m) = 1.0069\pm0.0070$, indicating there is no tension between early parameters and late parameters in the framework of $Λ$CDM model. As a result, the Hubble tension may arise from the differences of datasets or unknown systematic errors in the current data. In addition, we forecast the future BAO measurements of ${\rm ratio}(ω_m)$, using several galaxy redshift surveys and 21 cm intensity mapping surveys, and find that these measurements can significantly improve the precision of cosmological parameters.

astro-ph.CO↗

arXiv2024

Stability and regularization for ill-posed Cauchy problem of a stochastic parabolic differential equation

In this paper, we investigate an ill-posed Cauchy problem involving a stochastic parabolic equation. We first establish a Carleman estimate for this equation. Leveraging this estimate, we derive the conditional stability and convergence rate of the Tikhonov regularization method for the aforementioned ill-posed Cauchy problem. To complement our theoretical analysis, we employ kernel-based learning theory to implement the completed Tikhonov regularization method for several numerical examples.

math.NA↗

arXiv2024

SubGDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learning

Molecular representation learning has shown great success in advancing AI-based drug discovery. The core of many recent works is based on the fact that the 3D geometric structure of molecules provides essential information about their physical and chemical characteristics. Recently, denoising diffusion probabilistic models have achieved impressive performance in 3D molecular representation learning. However, most existing molecular diffusion models treat each atom as an independent entity, overlooking the dependency among atoms within the molecular substructures. This paper introduces a novel approach that enhances molecular representation learning by incorporating substructural information within the diffusion process. We propose a novel diffusion model termed SubGDiff for involving the molecular subgraph information in diffusion. Specifically, SubGDiff adopts three vital techniques: i) subgraph prediction, ii) expectation state, and iii) k-step same subgraph diffusion, to enhance the perception of molecular substructure in the denoising network. Experimentally, extensive downstream tasks demonstrate the superior performance of our approach. The code is available at https://github.com/youjibiying/SubGDiff.

cs.LG↗

arXiv2024

Coexisting Néel and charge density wave orders in attractive three-color fermions

In optical lattices attractive ultracold fermions with three hyperfine-spin components (colors) can form three fermionic configurations depending on interactions: unbound fermion, on-site trion and off-site trion, leading to the coexistence of multiple Fermi species in the ordered phase, which manifests that the attractive three-color fermions are unique from other correlated fermion systems and may host intriguing phases and phase transitions. At temperature below the superexchange energy scale, we employ the determinant quantum Monte Carlo (QMC) method to investigate the phases and phase transitions in the half-filled attractive three-color Hubbard model on a honeycomb lattice where Hubbard interactions are color-dependent (anisotropic interactions) and the coupling between color 3 and colors (1, 2) serves as a control parameter. In the coupling regime where on-site and off-site trions coexist, our QMC simulations demonstrate coexisting Néel and charge density wave orders which are common in condensed matter but rare in ultracold atoms. At weak coupling where the color superfluid (CSF) order is scattered by color-3 fermions, we find that very small coupling of color 3 with colors (1, 2) can destroy the CSF order and the vanishing of the CSF order is not immediately accompanied by the emergence of the on-site trionic phase, which strikingly disagrees with the prevalent results of dynamical mean-field theory. The underlying mechanisms of the coexisting charge/spin orders and the CSF order breaking are also presented based on intuitive physical pictures.

cond-mat.quant-gas↗

arXiv2024

Null controllability for stochastic fourth order semi-discrete parabolic equations

This paper is devoted to studying null controllability for a class of stochastic fourth order semi-discrete parabolic equations, where the spatial variable is discretized with finite difference scheme and the time is kept as a continuous variable. For this purpose, we establish a new global Carleman estimate for a backward stochastic fourth order semi-discrete parabolic operators, in which the large parameter is connected to the mesh size. A relaxed observability estimate is established for backward stochastic fourth order semi-discrete parabolic equations by this new Carleman estimate, with an explicit observability constant that depends on the discretization parameter and coefficients of lower order terms. Then, the $ϕ$-null controllability of the stochastic fourth order semi-discrete parabolic equations is proved using the standard duality technique.

math.OC↗

arXiv2024

A Dual Curriculum Learning Framework for Multi-UAV Pursuit-Evasion in Diverse Environments

This paper addresses multi-UAV pursuit-evasion, where a group of drones cooperates to capture a fast evader in a confined environment with obstacles. Existing heuristic algorithms, which simplify the pursuit-evasion problem, often lack expressive coordination strategies and struggle to capture the evader in extreme scenarios, such as when the evader moves at high speeds. In contrast, reinforcement learning (RL) has been applied to this problem and has the potential to obtain highly cooperative capture strategies. However, RL-based methods face challenges in training for complex 3-dimensional scenarios with diverse task settings due to the vast exploration space. The dynamics constraints of drones further restrict the ability of reinforcement learning to acquire high-performance capture strategies. In this work, we introduce a dual curriculum learning framework, named DualCL, which addresses multi-UAV pursuit-evasion in diverse environments and demonstrates zero-shot transfer ability to unseen scenarios. DualCL comprises two main components: the Intrinsic Parameter Curriculum Proposer, which progressively suggests intrinsic parameters from easy to hard to improve the capture capability of drones, and the External Environment Generator, tasked with exploring unresolved scenarios and generating appropriate training distributions of external environment parameters. The simulation experimental results show that DualCL significantly outperforms baseline methods, achieving over 90% capture rate and reducing the capture timestep by at least 27.5% in the training scenarios. Additionally, it exhibits the best zero-shot generalization ability in unseen environments. Moreover, we demonstrate the transferability of our pursuit strategy from simulation to real-world environments. Further details can be found on the project website at https://sites.google.com/view/dualcl.

cs.LG↗

arXiv2024

Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition

Semi-supervised action recognition aims to improve spatio-temporal reasoning ability with a few labeled data in conjunction with a large amount of unlabeled data. Albeit recent advancements, existing powerful methods are still prone to making ambiguous predictions under scarce labeled data, embodied as the limitation of distinguishing different actions with similar spatio-temporal information. In this paper, we approach this problem by empowering the model two aspects of capability, namely discriminative spatial modeling and temporal structure modeling for learning discriminative spatio-temporal representations. Specifically, we propose an Adaptive Contrastive Learning~(ACL) strategy. It assesses the confidence of all unlabeled samples by the class prototypes of the labeled data, and adaptively selects positive-negative samples from a pseudo-labeled sample bank to construct contrastive learning. Additionally, we introduce a Multi-scale Temporal Learning~(MTL) strategy. It could highlight informative semantics from long-term clips and integrate them into the short-term clip while suppressing noisy information. Afterwards, both of these two new techniques are integrated in a unified framework to encourage the model to make accurate predictions. Extensive experiments on UCF101, HMDB51 and Kinetics400 show the superiority of our method over prior state-of-the-art approaches.

cs.CV↗

arXiv2024

Floquet Nonadiabatic Mixed Quantum-Classical Dynamics in Laser-Dressed Solid Systems

In this paper, we introduce the Floquet Ehrenfest and Floquet surface hopping approaches to study the nonadiabatic dynamics in the laser-dressed solid systems. We demonstrate that these two approaches can be formulated in both real and reciprocal spaces. Using these approaches, we are able to simulate the interaction between electronic carriers and phonons under periodic drivings, such as strong light-matter interactions. Employing the Holstein and Peierls models, we show that the strong light-matter interactions can effectively modulate the dynamics of electronic population and mobility. Notably, our study demonstrates the feasibility and effectiveness of modeling low-momentum carriers' interactions with phonons using a truncated reciprocal-space basis, an approach impractical in real-space frameworks. Moreover, we reveal that even with significant truncation, carrier populations derived from surface hopping maintain greater accuracy compared to those obtained via meanfield dynamics. These results underscore the potential of our proposed methods in advancing the understanding of carrier-phonon interactions in various laser-dressed materials.

physics.chem-ph↗

arXiv2024

LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models

While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features. To address the SQA challenge on LLMs, we initially curated the free-form and open-ended LibriSQA dataset from Librispeech, comprising Part I with natural conversational formats and Part II encompassing multiple-choice questions followed by answers and analytical segments. Both parts collectively include 107k SQA pairs that cover various topics. Given the evident paucity of existing speech-text LLMs, we propose a lightweight, end-to-end framework to execute the SQA task on the LibriSQA, witnessing significant results. By reforming ASR into the SQA format, we further substantiate our framework's capability in handling ASR tasks. Our empirical findings bolster the LLMs' aptitude for aligning and comprehending multimodal information, paving the way for the development of universal multimodal LLMs. The dataset and demo can be found at https://github.com/ZihanZhaoSJTU/LibriSQA.

cs.CL↗

arXiv2024

EPIM: Efficient Processing-In-Memory Accelerators based on Epitome

The utilization of large-scale neural networks on Processing-In-Memory (PIM) accelerators encounters challenges due to constrained on-chip memory capacity. To tackle this issue, current works explore model compression algorithms to reduce the size of Convolutional Neural Networks (CNNs). Most of these algorithms either aim to represent neural operators with reduced-size parameters (e.g., quantization) or search for the best combinations of neural operators (e.g., neural architecture search). Designing neural operators to align with PIM accelerators' specifications is an area that warrants further study. In this paper, we introduce the Epitome, a lightweight neural operator offering convolution-like functionality, to craft memory-efficient CNN operators for PIM accelerators (EPIM). On the software side, we evaluate epitomes' latency and energy on PIM accelerators and introduce a PIM-aware layer-wise design method to enhance their hardware efficiency. We apply epitome-aware quantization to further reduce the size of epitomes. On the hardware side, we modify the datapath of current PIM accelerators to accommodate epitomes and implement a feature map reuse technique to reduce computation cost. Experimental results reveal that our 3-bit quantized EPIM-ResNet50 attains 71.59% top-1 accuracy on ImageNet, reducing crossbar areas by 30.65 times. EPIM surpasses the state-of-the-art pruning methods on PIM.

cs.AR↗

arXiv2024

Two system transformation data-driven algorithms for linear quadratic mean-field games

This paper studies a class of continuous-time linear quadratic (LQ) mean-field game problems. We develop two system transformation data-driven algorithms to approximate the decentralized strategies of the LQ mean-field games. The main feature of the obtained data-driven algorithms is that they eliminate the requirement on all system matrices. First, we transform the original stochastic system into an ordinary differential equation (ODE). Subsequently, we construct some Kronecker product-based matrices by the input/state data of the ODE. By virtue of these matrices, we implement a model-based policy iteration (PI) algorithm and a model-based value iteration (VI) algorithm in a data-driven fashion. In addition, we also demonstrate the convergence of these two data-driven algorithms under some mild conditions. Finally, we illustrate the practicality of our algorithms via two numerical examples.

math.OC↗

arXiv2024

A competitive game optimization algorithm for Unmanned Aerial Vehicle path planning

To solve the Unmanned Aerial Vehicle (UAV) path planning problem, a meta-heuristic optimization algorithm called competitive game optimizer (CGO) is proposed. In the CGO model, three phases of exploration and exploitation, and candidate replacement, are established, corresponding to the player's search for supplies and combat, and the movement toward a safe zone. In the algorithm exploration phase, Levy flight is introduced to improve the global convergence of the algorithm. The encounter probability which adaptively changes with the number of iterations is also introduced in the CGO. The balance between exploration and exploitation of solution space of optimization problem is realized, and each step is described and modeled mathematically. The performance of the CGO was evaluated on a set of 41 test functions taken from CEC2017 and CEC2022. It was then compared with eight widely recognized meta-heuristic optimization algorithms. The simulation results demonstrate that the proposed algorithm successfully achieves a balanced trade-off between exploration and exploitation, showcasing remarkable advantages when compared to seven classical algorithms. In addition, in order to further verify the effectiveness of the CGO, the CGO is applied to 8 practical engineering design problems and UAV path planning, and the results show that the CGO has strong performance in dealing with these practical optimization problems, and has a good application prospect.

eess.SY↗