arXiv subjects

Meng Li

Publications and source records attributed to Meng Li.

At least 235 records · Page 13Linked to original sources

arXiv2024

Online Learning of Multiple Tasks and Their Relationships : Testing on Spam Email Data and EEG Signals Recorded in Construction Fields

This paper examines an online multi-task learning (OMTL) method, which processes data sequentially to predict labels across related tasks. The framework learns task weights and their relatedness concurrently. Unlike previous models that assumed static task relatedness, our approach treats tasks as initially independent, updating their relatedness iteratively using newly calculated weight vectors. We introduced three rules to update the task relatedness matrix: OMTLCOV, OMTLLOG, and OMTLVON, and compared them against a conventional method (CMTL) that uses a fixed relatedness value. Performance evaluations on three datasets a spam dataset and two EEG datasets from construction workers under varying conditions demonstrated that our OMTL methods outperform CMTL, improving accuracy by 1% to 3% on EEG data, and maintaining low error rates around 12% on the spam dataset.

cs.LG↗

arXiv2024

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

cs.CL↗

arXiv2024

Structure-preserving weighted BDF2 methods for Anisotropic Cahn-Hilliard model: uniform/variable-time-steps

In this paper, we innovatively develop uniform/variable-time-step weighted and shifted BDF2 (WSBDF2) methods for the anisotropic Cahn-Hilliard (CH) model, combining the scalar auxiliary variable (SAV) approach with two types of stabilized techniques. Using the concept of $G$-stability, the uniform-time-step WSBDF2 method is theoretically proved to be energy-stable. Due to the inapplicability of the relevant G-stability properties, another technique is adopted in this work to demonstrate the energy stability of the variable-time-step WSBDF2 method. In addition, the two numerical schemes are all mass-conservative.Finally, numerous numerical simulations are presented to demonstrate the stability and accuracy of these schemes.

math.NA↗

arXiv2024

Insect Diversity Estimation in Polarimetric Lidar

Identification of insects in flight is a particular challenge for ecologists in several settings with no other method able to count and classify insects at the pace of entomological lidar. Thus, it can play a unique role as a non-intrusive diagnostic tool to assess insect biodiversity, inform planning, and evaluate mitigation efforts aimed at tackling declines in insect abundance and diversity. While species richness of co-existing insects could reach tens of thousands, to date, photonic sensors and lidars can differentiate roughly one hundred signal types. This taxonomic specificity or number of discernible signal types is currently limited by instrumentation and algorithm sophistication. In this study we report 32,533 observations of wild flying insects along a 500-meter transect. We report the benefits of lidar polarization bands for differentiating species and compare the performance of two unsupervised clustering algorithms, namely Hierarchical Cluster Analysis and Gaussian Mixture Model. We demonstrate that polarimetric properties could be partially predicted even with unpolarized light, thus polarimetric lidar bands provide only a minor improvement in specificity. Finally, we use physical properties of the clustered observation, such as wing beat frequency, daily activity patterns, and spatial distribution, to establish a lower bound for the number of species represented by the differentiated signal types.

q-bio.QM↗

arXiv2024

Operators' cognitive performance under extreme hot-humid exposure and its physiological-psychological mechanism based on ECG, fNIRS, and Eye Tracking

Operators' cognitive functions are impaired significantly under extreme heat stress, potentially resulting in more severe secondary disasters. This research investigated the impact of elevated temperature and humidity (25 60%RH, 30 70%RH, 35 80%RH, 40 90%RH) on the cognitive functions and performance of operators. Meanwhile, we explored the psychological-physiological mechanism underlying the change in performance by electrocardiogram (ECG), functional near-infrared spectroscopy (fNIRS), and eye tracking physiologically. Psychological aspects such as situation awareness, workload, and working memory were assessed. Eventually, we verified and extended the maximal adaptability model to the extreme condition. Unexpectedly, a temporary improvement in simple reaction tasks but rapid impairment in advanced cognitive functions (i.e. situation awareness, communication, working memory) was obtained above 35 WBGT. The best performance in a suitable environment was due to more effective activation in the prefrontal cortex (PFC). With temperature increasing, more mistakes occurred and comprehension was impaired due to drowsiness and lower arousal levels, according to evidence of compensatory effect in fNIRS. In the extreme environment, the enhanced PFC cooperation with higher functional connectivity resulted in a temporary improvement, while depressed activation in PFC, heavy physical load, and poor regulation of the cardiovascular system restricted it. Our results provide a detailed study of the process of operators' performance and cognitive functions when encountering increasing heat stress, as well as its underlying mechanisms from a neuroergonomics perspective. This can contribute to a better understanding of the interaction between operators' performance and workplace conditions, and help to achieve a more reliable human-centered production system in the promising era of Industry 5.0.

q-bio.NC↗

arXiv2024

Structure-preserving parametric finite element methods for simulating axisymmetric solid-state dewetting problems with anisotropic surface energies

Solid-state dewetting (SSD), a widespread phenomenon in solid-solid-vapor system, could be used to describe the accumulation of solid thin films on the substrate. In this work, we consider the sharp interface model for axisymmetric SSD with anisotropic surface energy. By introducing two types of surface energy matrices from the anisotropy functions,we aim to design two structure-preserving algorithms for the axisymmetric SSD. The newly designed schemes are applicable to a broader range of anisotropy functions, and we can theoretically prove their volume conservation and energy stability. In addition, based on a novel weak formulation for the axisymmetric SSD, we further build another two numerical schemes that have good mesh properties. Finally, numerous numerical tests are reported to showcase the accuracy and efficiency of the numerical methods.

math.NA↗

arXiv2024

Structure-preserving finite element methods for computing dynamics of rotating Bose-Einstein condensate

This work is concerned with the construction and analysis of structure-preserving Galerkin methods for computing the dynamics of rotating Bose-Einstein condensate (BEC) based on the Gross-Pitaevskii equation with angular momentum rotation. Due to the presence of the rotation term, constructing finite element methods (FEMs) that preserve both mass and energy remains an unresolved issue, particularly in the context of nonconforming FEMs. Furthermore, in comparison to existing works, we provide a comprehensive convergence analysis, offering a thorough demonstration of the methods' optimal and high-order convergence properties. Finally, extensive numerical results are presented to check the theoretical analysis of the structure-preserving numerical method for rotating BEC, and the quantized vortex lattice's behavior is scrutinized through a series of numerical tests.

math.NA↗

arXiv2024

FastQuery: Communication-efficient Embedding Table Query for Private LLM Inference

With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic encryption (HE) has been proposed to protect user query privacy. However, a private embedding table query has to be formulated as a HE-based matrix-vector multiplication problem and suffers from enormous computation and communication overhead. We observe the overhead mainly comes from the neglect of 1) the one-hot nature of user queries and 2) the robustness of the embedding table to low bit-width quantization noise. Hence, in this paper, we propose a private embedding table query optimization framework, dubbed FastQuery. FastQuery features a communication-aware embedding table quantization algorithm and a one-hot-aware dense packing algorithm to simultaneously reduce both the computation and communication costs. Compared to prior-art HE-based frameworks, e.g., Cheetah, Iron, and Bumblebee, FastQuery achieves more than $4.3\times$, $2.7\times$, $1.3\times$ latency reduction, respectively and more than $75.7\times$, $60.2\times$, $20.2\times$ communication reduction, respectively, on both LLAMA-7B and LLAMA-30B.

cs.CR↗

arXiv2024

Stochastic Multivariate Universal-Radix Finite-State Machine: a Theoretically and Practically Elegant Nonlinear Function Approximator

Nonlinearities are crucial for capturing complex input-output relationships especially in deep neural networks. However, nonlinear functions often incur various hardware and compute overheads. Meanwhile, stochastic computing (SC) has emerged as a promising approach to tackle this challenge by trading output precision for hardware simplicity. To this end, this paper proposes a first-of-its-kind stochastic multivariate universal-radix finite-state machine (SMURF) that harnesses SC for hardware-simplistic multivariate nonlinear function generation at high accuracy. We present the finite-state machine (FSM) architecture for SMURF, as well as analytical derivations of sampling gate coefficients for accurately approximating generic nonlinear functions. Experiments demonstrate the superiority of SMURF, requiring only 16.07% area and 14.45% power consumption of Taylor-series approximation, and merely 2.22% area of look-up table (LUT) schemes.

cs.LG↗

arXiv2024

vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs

IoT devices based on microcontroller units (MCU) provide ultra-low power consumption and ubiquitous computation for near-sensor deep learning models (DNN). However, the memory of MCU is usually 2-3 orders of magnitude smaller than mobile devices, which makes it challenging to map DNNs onto MCUs. Previous work separates memory management and kernel implementation for MCU and relies on coarse-grained memory management techniques such as inplace update to reduce memory consumption. In this paper, we propose to coordinate memory management and kernel optimization for DNN inference on MCUs to enable fine-grained memory management. The key idea is to virtualize the limited memory of MCU as a large memory pool. Each kernel divides the memory pool into kernel-specific segments and handles segment load and store while computing DNN layers. Memory consumption can be reduced because using the fine-grained segment-level memory control, we can overlap the memory footprint of different tensors without the need to materialize them at the same time. Following this idea, we implement \ours{} for DNN inference on MCU. Evaluation for single layers on ARM Cortex-M4 and Cortex-M7 processors shows that \ours{} can reduce from $12.0\%$ to $49.5\%$ RAM usage and from $20.6\%$ to $53.0\%$ energy consumption compared to state-of-the-art work. For full DNN evaluation, \ours{} can reduce the memory bottleneck by $61.5\%$, enabling more models to be deployed on low-end MCUs.

cs.AR↗

arXiv2024

EQO: Exploring Ultra-Efficient Private Inference with Winograd-Based Protocol and Quantization Co-Optimization

Private convolutional neural network (CNN) inference based on secure two-party computation (2PC) suffers from high communication and latency overhead, especially from convolution layers. In this paper, we propose EQO, a quantized 2PC inference framework that jointly optimizes the CNNs and 2PC protocols. EQO features a novel 2PC protocol that combines Winograd transformation with quantization for efficient convolution computation. However, we observe naively combining quantization and Winograd convolution is sub-optimal: Winograd transformations introduce extensive local additions and weight outliers that increase the quantization bit widths and require frequent bit width conversions with non-negligible communication overhead. Therefore, at the protocol level, we propose a series of optimizations for the 2PC inference graph to minimize the communication. At the network level, We develop a sensitivity-based mixed-precision quantization algorithm to optimize network accuracy given communication constraints. We further propose a 2PC-friendly bit re-weighting algorithm to accommodate weight outliers without increasing bit widths. With extensive experiments, EQO demonstrates 11.7x, 3.6x, and 6.3x communication reduction with 1.29%, 1.16%, and 1.29% higher accuracy compared to state-of-the-art frameworks SiRNN, COINN, and CoPriv, respectively.

cs.CR↗

arXiv2024

Temporal-Spatial Manipulation of Bi-Focal Bi-Chromatic Fields for Terahertz Radiations

Mixing the fundamental ($ω$) and the second harmonic (2$ω$) waves in gas phase is a widely employed technique for emitting terahertz (THz) pulses. The THz generation driven by bi-chromatic fields can be described by the photocurrent model, where the THz generation is attributed to free electrons ionized by the $ω$ field, and the 2$ω$ field provides a perturbation to break the symmetry of the asymptotic momentum of free electrons. However, we find that the THz radiation is amplified by one order of magnitude when driven by bi-focal bi-chromatic fields, contradicting the common understanding of the photocurrent model. Meanwhile, present measurements demonstrate that the THz radiation mainly originates from the plasma created by the 2$ω$ pulses instead of the $ω$ pulses. Energy transfer from the 2$ω$ beam to the THz beam during the THz generation has been observed, validating the major contribution of the 2$ω$ beam. Furthermore, the THz bandwidth has been observed to extensively exceed the bandwidth of the pump pulse, not be explained by the photocurrent model as well. These counterintuitive results indicate that undiscovered physical mechanisms are involved in bi-chromatic THz generation in plasma, presenting a significant challenge for understanding strong-field nonlinear optics and simultaneously expanding various applications.

physics.optics↗

arXiv2024

A neuroergonomics model to evaluating nuclear power plants operators' performance under heat stress driven by ECG time-frequency spectrums and fNIRS prefrontal cortex network: a CNN-GAT fusion model

Operators experience complicated physiological and psychological states when exposed to extreme heat stress, which can impair cognitive function and decrease performance significantly, ultimately leading to severe secondary disasters. Therefore, there is an urgent need for a feasible technique to identify their abnormal states to enhance the reliability of human-cybernetics systems. With the advancement of deep learning in physiological modeling, a model for evaluating operators' performance driven by electrocardiogram (ECG) and functional near-infrared spectroscopy (fNIRS) was proposed, demonstrating high ecological validity. The model fused a convolutional neural network (CNN) backbone and a graph attention network (GAT) backbone to extract discriminative features from ECG time-frequency spectrums and fNIRS prefrontal cortex (PFC) network respectively with deeper neuroscience domain knowledge, and eventually achieved 0.90 AUC. Results supported that handcrafted features extracted by specialized neuroscience methods can alleviate overfitting. Inspired by the small-world nature of the brain network, the fNIRS PFC network was organized as an undirected graph and embedded by GAT. It is proven to perform better in information aggregation and delivery compared to a simple non-linear transformation. The model provides a potential neuroergonomics application for evaluating the human state in vital human-cybernetics systems under industry 5.0 scenarios.

cs.HC↗

arXiv2024

Structure-preserving, weighted implicit-explicit schemes for multi-phase incompressible Navier-Stokes/Darcy coupled nonlocal Allen-Cahn model

A multitude of substances exist as mixtures comprising multiple chemical components in the natural world. These substances undergo morphological changes under external influences. the phase field model coupled with fluid flow, the dynamic movement and evolution of the phase interface intricately interact with the fluid motion. This article focuses on the N-component models that couple the conservative Allen-Cahn equation with two types of incompressible fluid flow systems: the Navier-Stokes equation and the Darcy equation. By utilizing the scalar auxiliary variable method and the projection method, we innovatively construct two types of structure-preserving weighted implicit-explicit schemes for the coupled models, resulting in fully decoupled linear systems and second-order accuracy in time. The schemes are proved to be mass-conservative. In addition, with the application of $G$-norm inspired by the idea of $G$-stability, we rigorously establish its unconditional energy stability. Finally, the performance of the proposed scheme is verified by some numerical simulations.

math.NA↗

arXiv2024

Ubiquitous short-range order in multi-principal element alloys

Recent research in multi-principal element alloys (MPEAs) has increasingly focused on the exploration and exploitation of short-range order (SRO) to enhance material performance. However, the understanding of SRO formation and the precise tuning of it within MPEAs remains poorly understood, limiting the comprehension of its impact on material properties and impeding the advancement of SRO engineering. Here, leveraging advanced additive manufacturing techniques that produce samples with a wide range of cooling rates (up to 10^7 K/s) and an improved quantitative electron microscopy method, we characterize SRO in three CoCrNi-based MPEAs to unravel the role of processing route and thermal history on SRO. Surprisingly, irrespective of the processing and thermal treatment applied, all samples exhibit similar levels of SRO, suggesting that prevalent SRO may form during the solidification process. Atomistic simulations of solidification verify that local chemical ordering arises in the liquid-solid interface (solidification front) even under the extreme cooling rate of 10^11 K/s. This phenomenon stems from the swift atomic diffusion in the supercooled liquid, which matches or even surpasses the rate of solidification. Therefore, SRO is an inherent characteristic of most MPEAs, insensitive to variations in cooling rates and annealing treatments typically available in experiments. Integrating thermal treatment with other strategies, such as mechanical deformation and irradiation, might be more effective approaches for harnessing SRO to achieve controlled material properties.

cond-mat.mtrl-sci↗

arXiv2024

ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding

Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is hampered by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verification, e.g., Medusa, has been proposed to improve decoding parallelism and efficiency, it often struggles with maintaining contextual relationships due to its independent token prediction approach and incurs significant verification overhead, especially with large tree sizes and batch processing. In this paper, we propose ProPD, an efficient LLM parallel decoding framework based on dynamic token tree pruning and generation. ProPD features an advanced early pruning mechanism to efficiently eliminate unpromising token sequences to improve verification efficiency. Additionally, it introduces a dynamic token tree generation algorithm to balance the computation and parallelism of the verification phase in real-time and maximize the overall efficiency across different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD consistently outperforms existing decoding algorithms by 1.1-3.2x.

cs.LG↗

arXiv2024

ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer

Stochastic computing (SC) has emerged as a promising computing paradigm for neural acceleration. However, how to accelerate the state-of-the-art Vision Transformer (ViT) with SC remains unclear. Unlike convolutional neural networks, ViTs introduce notable compatibility and efficiency challenges because of their nonlinear functions, e.g., softmax and Gaussian Error Linear Units (GELU). In this paper, for the first time, a ViT accelerator based on end-to-end SC, dubbed ASCEND, is proposed. ASCEND co-designs the SC circuits and ViT networks to enable accurate yet efficient acceleration. To overcome the compatibility challenges, ASCEND proposes a novel deterministic SC block for GELU and leverages an SC-friendly iterative approximate algorithm to design an accurate and efficient softmax circuit. To improve inference efficiency, ASCEND develops a two-stage training pipeline to produce accurate low-precision ViTs. With extensive experiments, we show the proposed GELU and softmax blocks achieve 56.3% and 22.6% error reduction compared to existing SC designs, respectively and reduce the area-delay product (ADP) by 5.29x and 12.6x, respectively. Moreover, compared to the baseline low-precision ViTs, ASCEND also achieves significant accuracy improvements on CIFAR10 and CIFAR100.

eess.SY↗

arXiv2024

HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference

Secure two-party computation with homomorphic encryption (HE) protects data privacy with a formal security guarantee but suffers from high communication overhead. While previous works, e.g., Cheetah, Iron, etc, have proposed efficient HE-based protocols for different neural network (NN) operations, they still assume high precision, e.g., fixed point 37 bit, for the NN operations and ignore NNs' native robustness against quantization error. In this paper, we propose HEQuant, which features low-precision-quantization-aware optimization for the HE-based protocols. We observe the benefit of a naive combination of quantization and HE quickly saturates as bit precision goes down. Hence, to further improve communication efficiency, we propose a series of optimizations, including an intra-coefficient packing algorithm and a quantization-aware tiling algorithm, to simultaneously reduce the number and precision of the transferred data. Compared with prior-art HE-based protocols, e.g., CrypTFlow2, Cheetah, Iron, etc, HEQuant achieves $3.5\sim 23.4\times$ communication reduction and $3.0\sim 9.3\times$ latency reduction. Meanwhile, when compared with prior-art network optimization frameworks, e.g., SENet, SNL, etc, HEQuant also achieves $3.1\sim 3.6\times$ communication reduction.

cs.CR↗