arXiv subjects

Yu Wang

Publications and source records attributed to Yu Wang.

At least 703 records · Page 39Linked to original sources

arXiv2024

ΩSFormer: Dual-Modal Ω-like Super-Resolution Transformer Network for Cross-scale and High-accuracy Terraced Field Vectorization Extraction

Terraced field is a significant engineering practice for soil and water conservation (SWC). Terraced field extraction from remotely sensed imagery is the foundation for monitoring and evaluating SWC. This study is the first to propose a novel dual-modal Ω-like super-resolution Transformer network for intelligent TFVE, offering the following advantages: (1) reducing edge segmentation error from conventional multi-scale downsampling encoder, through fusing original high-resolution features with downsampling features at each step of encoder and leveraging a multi-head attention mechanism; (2) improving the accuracy of TFVE by proposing a Ω-like network structure, which fully integrates rich high-level features from both spectral and terrain data to form cross-scale super-resolution features; (3) validating an optimal fusion scheme for cross-modal and cross-scale (i.e., inconsistent spatial resolution between remotely sensed imagery and DEM) super-resolution feature extraction; (4) mitigating uncertainty between segmentation edge pixels by a coarse-to-fine and spatial topological semantic relationship optimization (STSRO) segmentation strategy; (5) leveraging contour vibration neural network to continuously optimize parameters and iteratively vectorize terraced fields from semantic segmentation results. Moreover, a DMRVD for deep-learning-based TFVE was created for the first time, which covers nine study areas in four provinces of China, with a total coverage area of 22441 square kilometers. To assess the performance of ΩSFormer, classic and SOTA networks were compared. The mIOU of ΩSFormer has improved by 0.165, 0.297 and 0.128 respectively, when compared with best accuracy single-modal remotely sensed imagery, single-modal DEM and dual-modal result.

cs.CV↗

arXiv2024

LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization

Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. To address this issue, we conducted analysis and identified significant redundancy in three dimensions of the VAR model: (1) the attention map, (2) the attention outputs when using classifier free guidance, and (3) the data precision. Correspondingly, we proposed efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance. With negligible performance lost (less than 0.056 FID increase), we could achieve 85.2% reduction in attention computation, 50% reduction in overall memory and 1.5x latency reduction. To ensure deployment feasibility, we developed efficient training-free compression techniques and analyze the deployment feasibility and efficiency gain of each technique.

cs.CV↗

arXiv2024

Glo-In-One-v2: Holistic Identification of Glomerular Cells, Tissues, and Lesions in Human and Mouse Histopathology

Segmenting glomerular intraglomerular tissue and lesions traditionally depends on detailed morphological evaluations by expert nephropathologists, a labor-intensive process susceptible to interobserver variability. Our group previously developed the Glo-In-One toolkit for integrated detection and segmentation of glomeruli. In this study, we leverage the Glo-In-One toolkit to version 2 with fine-grained segmentation capabilities, curating 14 distinct labels for tissue regions, cells, and lesions across a dataset of 23,529 annotated glomeruli across human and mouse histopathology data. To our knowledge, this dataset is among the largest of its kind to date.In this study, we present a single dynamic head deep learning architecture designed to segment 14 classes within partially labeled images of human and mouse pathology data. Our model was trained using a training set derived from 368 annotated kidney whole-slide images (WSIs) to identify 5 key intraglomerular tissues covering Bowman's capsule, glomerular tuft, mesangium, mesangial cells, and podocytes. Additionally, the network segments 9 glomerular lesion classes including adhesion, capsular drop, global sclerosis, hyalinosis, mesangial lysis, microaneurysm, nodular sclerosis, mesangial expansion, and segmental sclerosis. The glomerulus segmentation model achieved a decent performance compared with baselines, and achieved a 76.5 % average Dice Similarity Coefficient (DSC). Additional, transfer learning from rodent to human for glomerular lesion segmentation model has enhanced the average segmentation accuracy across different types of lesions by more than 3 %, as measured by Dice scores. The Glo-In-One-v2 model and trained weight have been made publicly available at https: //github.com/hrlblab/Glo-In-One_v2.

eess.IV↗

arXiv2024

Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study

Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs. However, for humans, teaching improves not only students but also teachers, by fostering more rigorous and clear reasoning as well as knowledge building. We ask: Can LLMs also learn by teaching (LbT) for better reasoning? If the answer is yes, we can potentially unlock the possibility of continuously advancing the models without solely relying on human-produced data or stronger models. In this paper, we provide a preliminary exploration on this question. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and bring improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT: observing students' feedback, learning from the feedback, and learning iteratively, with the goals of improving answer accuracy without training or improving models' inherent capability with fine-tuning. We reveal some findings: (1) Teaching materials that make it easier for students to learn have clearer and more accurate logic when using in-context learning as the student's "learning" method; (2) Weak-to-strong generalization: LbT might help improve strong models by teaching weak models; (3) Diversity in students might help: teaching multiple students could be better than teaching one student or the teacher itself. We hope that our exploration can inspire future research on LbT and more broadly adopting the advanced techniques in education to improve LLMs. The code and website are at https://github.com/imagination-research/lbt and https://sites.google.com/view/llm-learning-by-teaching.

cs.CL↗

arXiv2024

Continual SFT Matches Multimodal RLHF with Negative Supervision

Multimodal RLHF usually happens after supervised finetuning (SFT) stage to continually improve vision-language models' (VLMs) comprehension. Conventional wisdom holds its superiority over continual SFT during this preference alignment stage. In this paper, we observe that the inherent value of multimodal RLHF lies in its negative supervision, the logit of the rejected responses. We thus propose a novel negative supervised finetuning (nSFT) approach that fully excavates these information resided. Our nSFT disentangles this negative supervision in RLHF paradigm, and continually aligns VLMs with a simple SFT loss. This is more memory efficient than multimodal RLHF where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required. The effectiveness of nSFT is rigorously proved by comparing it with various multimodal RLHF approaches, across different dataset sources, base VLMs and evaluation metrics. Besides, fruitful of ablations are provided to support our hypothesis. We hope this paper will stimulate further research to properly align large vision language models.

cs.LG↗

arXiv2024

Towards Accurate and Efficient Sub-8-Bit Integer Training

Neural network training is a memory- and compute-intensive task. Quantization, which enables low-bitwidth formats in training, can significantly mitigate the workload. To reduce quantization error, recent methods have developed new data formats and additional pre-processing operations on quantizers. However, it remains quite challenging to achieve high accuracy and efficiency simultaneously. In this paper, we explore sub-8-bit integer training from its essence of gradient descent optimization. Our integer training framework includes two components: ShiftQuant to realize accurate gradient estimation, and L1 normalization to smoothen the loss landscape. ShiftQuant attains performance that approaches the theoretical upper bound of group quantization. Furthermore, it liberates group quantization from inefficient memory rearrangement. The L1 normalization facilitates the implementation of fully quantized normalization layers with impressive convergence accuracy. Our method frees sub-8-bit integer training from pre-processing and supports general devices. This framework achieves negligible accuracy loss across various neural networks and tasks ($0.92\%$ on 4-bit ResNets, $0.61\%$ on 6-bit Transformers). The prototypical implementation of ShiftQuant achieves more than $1.85\times/15.3\%$ performance improvement on CPU/GPU compared to its FP16 counterparts, and $33.9\%$ resource consumption reduction on FPGA than the FP16 counterparts. The proposed fully-quantized L1 normalization layers achieve more than $35.54\%$ improvement in throughout on CPU compared to traditional L2 normalization layers. Moreover, theoretical analysis verifies the advancement of our method.

cs.LG↗

arXiv2024

WelQrate: Defining the Gold Standard in Small Molecule Drug Discovery Benchmarking

While deep learning has revolutionized computer-aided drug discovery, the AI community has predominantly focused on model innovation and placed less emphasis on establishing best benchmarking practices. We posit that without a sound model evaluation framework, the AI community's efforts cannot reach their full potential, thereby slowing the progress and transfer of innovation into real-world drug discovery. Thus, in this paper, we seek to establish a new gold standard for small molecule drug discovery benchmarking, WelQrate. Specifically, our contributions are threefold: WelQrate Dataset Collection - we introduce a meticulously curated collection of 9 datasets spanning 5 therapeutic target classes. Our hierarchical curation pipelines, designed by drug discovery experts, go beyond the primary high-throughput screen by leveraging additional confirmatory and counter screens along with rigorous domain-driven preprocessing, such as Pan-Assay Interference Compounds (PAINS) filtering, to ensure the high-quality data in the datasets; WelQrate Evaluation Framework - we propose a standardized model evaluation framework considering high-quality datasets, featurization, 3D conformation generation, evaluation metrics, and data splits, which provides a reliable benchmarking for drug discovery experts conducting real-world virtual screening; Benchmarking - we evaluate model performance through various research questions using the WelQrate dataset collection, exploring the effects of different models, dataset quality, featurization methods, and data splitting strategies on the results. In summary, we recommend adopting our proposed WelQrate as the gold standard in small molecule drug discovery benchmarking. The WelQrate dataset collection, along with the curation codes, and experimental scripts are all publicly available at WelQrate.org.

cs.LG↗

arXiv2024

Dynamics of relativistic vortex electrons in external laser fields

Investigating the interactions of vortex electrons with electromagnetic fields is crucial for advancing particle acceleration techniques, scattering theory in background fields, and developing novel electron beams for material diagnostics. In this work, we systematically study the dynamics of relativistic vortex electrons during their head-on collisions with linearly polarized (LP) and circularly polarized (CP) laser pulses, as well as their superposition. We develop a theoretical framework using Volkov-Bessel wave functions to describe the spatiotemporal characteristics of vortex electrons in these external fields. We show that the beam center of the vortex electron follows the classical trajectory of a point-charge electron while maintaining the transverse structure of both vortex eigenstates and superposition states. Specifically, CP laser pulses cause the beam center to rotate, while LP laser pulses induce a lateral shift. The combined effect of LP and CP laser pulses in a two-mode field results in a twisted spiral pattern. Our findings demonstrate the potential for versatile control of vortex electron beams using various laser modes, providing a foundation for future experimental and theoretical studies. This work serves as a benchmark reference for investigations into the manipulation of vortex electron beams using more realistic laser or other types of external fields.

physics.optics↗

arXiv2024

Unbiased approximation of the ergodic measure for piecewise $α$-stable Ornstein-Uhlenbeck processes arising in queueing networks

Piecewise $α$-stable Ornstein-Uhlenbeck (OU) processes arising in queue networks usually do not have an explicit dissipation, which makes the related numerical methods such as Euler-Maruyama (EM) scheme more difficult to analyze. We develop an EM scheme with decreasing step size $Λ=(η_n)_{n\in \mathbb{N}}$ to approximate their ergodic measures. This approximation does not have a bias and has a rate $η^{1/α}_n$ in Wasserstein-1 distance. We show by the classical OU process that our convergence rate is optimal. We further prove the central limit theorem (CLT) and moderate derivation principle (MDP) for the empirical measure of these piecewise $α$-stable Ornstein-Uhlenbeck processes. In addition, we use the Sinkhorn--Knopp algorithm to compute the Wasserstein-1 distance and conduct simulations for several concrete examples.

math.PR↗

arXiv2024

GRS-QA -- Graph Reasoning-Structured Question Answering Dataset

Large Language Models (LLMs) have excelled in multi-hop question-answering (M-QA) due to their advanced reasoning abilities. However, the impact of the inherent reasoning structures on LLM M-QA performance remains unclear, largely due to the absence of QA datasets that provide fine-grained reasoning structures. To address this gap, we introduce the Graph Reasoning-Structured Question Answering Dataset (GRS-QA), which includes both semantic contexts and reasoning structures for QA pairs. Unlike existing M-QA datasets, where different reasoning structures are entangled together, GRS-QA explicitly captures intricate reasoning pathways by constructing reasoning graphs, where nodes represent textual contexts and edges denote logical flows. These reasoning graphs of different structures enable a fine-grained evaluation of LLM reasoning capabilities across various reasoning structures. Our empirical analysis reveals that LLMs perform differently when handling questions with varying reasoning structures. This finding facilitates the exploration of textual structures as compared with semantics.

cs.CL↗

arXiv2024

Inverse problems for stochastic partial differential equations

This book aims to provide a brief overview of recent advancements in the theory of inverse problems for stochastic partial differential equations. In order to keep the content concise, we will only discuss the inverse problems of two typical classes of stochastic partial differential equations: second-order stochastic parabolic equations and secondorder stochastic hyperbolic equations. The main tool for studying these inverse problem is Carleman estimate. We do not intend to pursue any general treatment of the Carleman estimates themselves and choose direct arguments based on basic stochastic calculus, rather than more general sophisticated methods. As this field is still developing and there are many challenging issues to be addressed, the purpose of this book is not to serve as a comprehensive summary, but rather to spark interest and encourage further exploration in this area among readers. We prefer to present results that, from our perspective, include fresh and promising ideas. In cases where a complete mathematical theory is lacking, we only provide the available results. We do not intend for the current book to be encyclopedic in any sense, and the references are limited.

math.PR↗

arXiv2024

Selecting Between BERT and GPT for Text Classification in Political Science Research

Political scientists often grapple with data scarcity in text classification. Recently, fine-tuned BERT models and their variants have gained traction as effective solutions to address this issue. In this study, we investigate the potential of GPT-based models combined with prompt engineering as a viable alternative. We conduct a series of experiments across various classification tasks, differing in the number of classes and complexity, to evaluate the effectiveness of BERT-based versus GPT-based models in low-data scenarios. Our findings indicate that while zero-shot and few-shot learning with GPT models provide reasonable performance and are well-suited for early-stage research exploration, they generally fall short - or, at best, match - the performance of BERT fine-tuning, particularly as the training set reaches a substantial size (e.g., 1,000 samples). We conclude by comparing these approaches in terms of performance, ease of use, and cost, providing practical guidance for researchers facing data limitations. Our results are particularly relevant for those engaged in quantitative text analysis in low-resource settings or with limited labeled data.

cs.CL↗

arXiv2024

Robust estimation for high-dimensional time series with heavy tails

We study in this paper the problem of least absolute deviation (LAD) regression for high-dimensional heavy-tailed time series which have finite $α$-th moment with $α\in (1,2]$. To handle the heavy-tailed dependent data, we propose a Catoni type truncated minimization problem framework and obtain an $\mathcal{O}\big( \big( (d_1+d_2) (d_1\land d_2) \log^2 n / n \big)^{(α- 1)/α} \big)$ order excess risk, where $d_1$ and $d_2$ are the dimensionality and $n$ is the number of samples. We apply our result to study the LAD regression on high-dimensional heavy-tailed vector autoregressive (VAR) process. Simulations for the VAR($p$) model show that our new estimator with truncation are essential because the risk of the classical LAD has a tendency to blow up. We further apply our estimation to the real data and find that ours fits the data better than the classical LAD.

math.ST↗

arXiv2024

Extremal spectral radius and $g$-good $r$-component connectivity

For $F\subseteq V(G)$, if $G-F$ is a disconnected graph with at least $r$ components and each vertex $v\in V(G)\backslash F$ has at least $g$ neighbors, then $F$ is called a $g$-good $r$-component cut of $G$. The $g$-good $r$-component connectivity of $G$, denoted by $cκ_{g,r}(G)$, is the minimum cardinality of $g$-good $r$-component cuts of $G$. Let $\mathcal{G}_n^{k,δ}$ be the set of graphs of order $n$ with minimum degree $δ$ and $g$-good $r$-component connectivity $cκ_{g,r}(G)=k$. In the paper, we determine the extremal graphs attaining the maximum spectral radii among all graphs in $\mathcal{G}_n^{k,δ}$. A subset $F\subseteq V(G)$ is called a $g$-good neighbor cut of $G$ if $G-F$ is disconnected and each vertex $v\in V(G)\backslash F$ has at least $g$ neighbors. The $g$-good neighbor connectivity $κ_g(G)$ of a graph $G$ is the minimum cardinality of $g$-good neighbor cuts of $G$. The condition of $g$-good neighbor connectivity is weaker than that of $g$-good $r$-component connectivity, and there is no requirement on the number of components. As a counterpart, we also study similar problem for $g$-good neighbor connectivity.

math.CO↗

arXiv2024

Efficient and Effective Retrieval of Dense-Sparse Hybrid Vectors using Graph-based Approximate Nearest Neighbor Search

ANNS for embedded vector representations of texts is commonly used in information retrieval, with two important information representations being sparse and dense vectors. While it has been shown that combining these representations improves accuracy, the current method of conducting sparse and dense vector searches separately suffers from low scalability and high system complexity. Alternatively, building a unified index faces challenges with accuracy and efficiency. To address these issues, we propose a graph-based ANNS algorithm for dense-sparse hybrid vectors. Firstly, we propose a distribution alignment method to improve accuracy, which pre-samples dense and sparse vectors to analyze their distance distribution statistic, resulting in a 1%$\sim$9% increase in accuracy. Secondly, to improve efficiency, we design an adaptive two-stage computation strategy that initially computes dense distances only and later computes hybrid distances. Further, we prune the sparse vectors to speed up the calculation. Compared to naive implementation, we achieve $\sim2.1\times$ acceleration. Thorough experiments show that our algorithm achieves 8.9x$\sim$11.7x throughput at equal accuracy compared to existing hybrid vector search algorithms.

cs.IR↗

arXiv2024

A Compact Readout Electronics Based on Current Amplifier for Micromegas Detector in Muon Imaging

Muon imaging technology is an innovative imaging technique that can be applied in volcano imaging, heavy nuclear material detection, and archaeological research. The Micromegas detector is a promising choice for muon imaging due to its high spatial resolution and large area. However, the large number of readout channels poses a challenge for electronics. In this paper, a compact front-end electronics (FEE) for reading Micromegas detectors is presented. The electronics use the commercial current-to-digital readout chip, ADAS1128, which integrates 128 current amplifiers for multi-channel charge measurement. After verifying the basic performance of the electronics, the energy resolution was obtained with a radioactive source. Furthermore, a muon imaging system prototype was set up and its spatial resolution was evaluated in a test with cosmic ray muons. The system prototype can reconstruct the boundaries of sufficiently massive objects with a size of 2 cm in a scattering imaging test.

physics.ins-det↗

arXiv2024

A Survey of Small Language Models

Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models.

cs.CL↗

arXiv2024

Beyond the Phase Ordering Problem: Finding the Globally Optimal Code w.r.t. Optimization Phases

In this paper, we propose a new concept called \textit{semantically equivalence} \wrt \textit{optimization phases} \textit{(\sep)}, which defines the set of programs a compiler considers semantically equivalent to the input using a set of optimization phases. We show both theoretically and empirically that solving the phase ordering problem does not necessarily result in the most efficient code among all programs that a compiler deems semantically equivalent to the input, hereinafter referred to as the global optimal code \wrt optimization phases. To find the global optimal code \wrt optimization phases, we present a conceptual framework, leveraging the reverse of existing optimization phases. In theory, we prove that the framework is capable of finding the global optimal code for any program. We realize this framework into a technique, called \textit{iterative bi-directional optimization (\tool)}, which performs both the normal and reverse optimizations to increase and decrease the efficiency of the generated code, respectively. We evaluate \tool on C/C++ files randomly extracted from highly mature and influential programs (\eg, Linux kernel, OpenSSL, Z3). Results show that \tool frequently generates more efficient code -- measured by either code size or runtime performance -- than exhaustive search, which is the solution to the phase ordering problem. We also find by simply incorporating \tool's reverse optimization phases, the effectiveness of the optimization of state-of-the-art compilers (\eg, GCC/LLVM) can be significantly improved.

cs.PL↗