arXiv subjects

Lei Wang

Publications and source records attributed to Lei Wang.

At least 595 records · Page 33Linked to original sources

arXiv2024

YuLan: An Open-source Large Language Model

Large language models (LLMs) have become the foundation of many applications, leveraging their extensive capabilities in processing and understanding natural language. While many open-source LLMs have been released with technical reports, the lack of training details hinders further research and development. This paper presents the development of YuLan, a series of open-source LLMs with $12$ billion parameters. The base model of YuLan is pre-trained on approximately $1.7$T tokens derived from a diverse corpus, including massive English, Chinese, and multilingual texts. We design a three-stage pre-training method to enhance YuLan's overall capabilities. Subsequent phases of training incorporate instruction-tuning and human alignment, employing a substantial volume of high-quality synthesized data. To facilitate the learning of complex and long-tail knowledge, we devise a curriculum-learning framework throughout across these stages, which helps LLMs learn knowledge in an easy-to-hard manner. YuLan's training is finished on Jan, 2024 and has achieved performance on par with state-of-the-art LLMs across various English and Chinese benchmarks. This paper outlines a comprehensive technical roadmap for developing LLMs from scratch. Our model and codes are available at https://github.com/RUC-GSAI/YuLan-Chat.

cs.CL↗

arXiv2024

Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other

Emergent Large Language Models (LLMs) use their extraordinary performance and powerful deduction capacity to discern from traditional language models. However, the expenses of computational resources and storage for these LLMs are stunning, quantization then arises as a trending conversation. To address accuracy decay caused by quantization, two streams of works in post-training quantization methods stand out. One uses other weights to compensate existing quantization error, while the other transfers the quantization difficulty to other parts in the model. Combining both merits, we introduce Learnable Singular value Increment (LSI) as an advanced solution. LSI uses Singular Value Decomposition to extract singular values of the weights and make them learnable to help weights compensate each other conditioned on activation. Incorporating LSI with existing techniques, we achieve state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios. By unleashing the potential of LSI, efficient finetuning on quantized model is no longer a prohibitive problem.

cs.CL↗

arXiv2024

Direct Production of Light Scalar in the Type-I Two-Higgs-Doublet Model at the Lifetime Frontier of LHC

A light pseudoscalar $A$ in the sufficient large $\tanβ$ region of type-I two-Higgs-doublet model (2HDM) can be naturally a long-lived particle (LLP). We focus on $H^{\pm}A$, $HA$ and $AA$ pair productions via the electroweak processes mediated by the bosons at the LHC, including $pp \rightarrow W^\pm/Z \rightarrow H^{\pm}/H A$ and $pp \rightarrow h \rightarrow AA$ at the 14 TeV LHC. The possibility of probing $A$ as a LLP at the FASER-2, FACET, MoEDAL-MAPP-2, MATHUSLA is discussed. We find that FASER-2 fails to probe any parameter space within 0.2 GeV $< m_A <$ 10 GeV for all the considered processes. For 130 $< m_{H\pm} = m_H <$ 400 GeV, FACET, MoEDAL-MAPP-2 and MATHUSLA can probe $\tan β\lesssim 10^{4-6}$ for $m_A \lesssim 3$ GeV, and $\tan β\lesssim 10^{6-8}$ for 3 GeV $\lesssim m_A <$ 10 GeV from $pp \rightarrow W^\pm/Z \rightarrow H^{\pm}/Z A$ processes. And $pp \rightarrow h \rightarrow AA$ process covers similar parameter space. All processes can surpass the current limits.

hep-ph↗

arXiv2024

STEMO: Early Spatio-temporal Forecasting with Multi-Objective Reinforcement Learning

Accuracy and timeliness are indeed often conflicting goals in prediction tasks. Premature predictions may yield a higher rate of false alarms, whereas delaying predictions to gather more information can render them too late to be useful. In applications such as wildfires, crimes, and traffic jams, timely forecasting are vital for safeguarding human life and property. Consequently, finding a balance between accuracy and timeliness is crucial. In this paper, we propose an early spatio-temporal forecasting model based on Multi-Objective reinforcement learning that can either implement an optimal policy given a preference or infer the preference based on a small number of samples. The model addresses two primary challenges: 1) enhancing the accuracy of early forecasting and 2) providing the optimal policy for determining the most suitable prediction time for each area. Our method demonstrates superior performance on three large-scale real-world datasets, surpassing existing methods in early spatio-temporal forecasting tasks.

cs.LG↗

arXiv2024

Analysis and Evaluation of Kinect-based Action Recognition Algorithms

Human action recognition still exists many challenging problems such as different viewpoints, occlusion, lighting conditions, human body size and the speed of action execution, although it has been widely used in different areas. To tackle these challenges, the Kinect depth sensor has been developed to record real time depth sequences, which are insensitive to the color of human clothes and illumination conditions. Many methods on recognizing human action have been reported in the literature such as HON4D, HOPC, RBD and HDG, which use the 4D surface normals, pointclouds, skeleton-based model and depth gradients respectively to capture discriminative information from depth videos or skeleton data. In this research project, the performance of four aforementioned algorithms will be analyzed and evaluated using five benchmark datasets, which cover challenging issues such as noise, change of viewpoints, background clutters and occlusions. We also implemented and improved the HDG algorithm, and applied it in cross-view action recognition using the UWA3D Multiview Activity dataset. Moreover, we used different combinations of individual feature vectors in HDG for performance evaluation. The experimental results show that our improvement of HDG outperforms other three state-of-the-art algorithms for cross-view action recognition.

cs.CV↗

arXiv2024

Neural canonical transformations for vibrational spectra of molecules

The behavior of polyatomic molecules around their equilibrium positions can be regarded as quantum coupled anharmonic oscillators. Solving the corresponding Schrödinger equations can interpret or predict experimental spectra of molecules. In this study, we develop a novel approach to solve excited states of anharmonic vibrational systems. The normal coordinates of molecules are transformed into new coordinates through a normalizing flow parameterized by a neural network, facilitating the construction of a set of orthogonal many-body variational wavefunctions. Our methodology has been validated on an exactly solvable $64$-dimensional coupled harmonic oscillator, yielding numerical results with a relative error on the order of $10^{-6}$. Furthermore, the neural canonical transformations are also applied to calculate the energy levels of two specific molecules, acetonitrile ($\text{C}\text{H}_3\text{CN}$) and ethylene oxide ($\text{C}_2\text{H}_4\text{O}$) involving $12$ and $15$ vibrational modes respectively, which are in agreement with experimental findings. One of the key advantages of our approach is its flexibility concerning the potential energy surface, requiring no specific form for its application. Moreover, this method can be easily implemented on large-scale distributed computing platforms, making it particularly suitable for investigating more complex vibrational structures.

physics.chem-ph↗

arXiv2024

NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models

Numeral systems and units of measurement are two conjoined topics in activities of human beings and have mutual effects with the languages expressing them. Currently, the evaluation of Large Language Models (LLMs) often involves mathematical reasoning, yet little attention is given to how minor changes in numbers or units can drastically alter the complexity of problems and the performance of LLMs. In this paper, we scrutinize existing LLMs on processing of numerals and units of measurement by constructing datasets with perturbations. We first anatomize the reasoning of math word problems to different sub-procedures like numeral conversions from language to numbers and measurement conversions based on units. Then we further annotate math word problems from ancient Chinese arithmetic works which are challenging in numerals and units of measurement. Experiments on perturbed datasets demonstrate that LLMs still encounter difficulties in handling numeral and measurement conversions.

cs.CL↗

arXiv2024

A novel measurement method for SiPM external crosstalk probability at low temperature

Silicon photomultipliers (SiPMs) are being considered as potential replacements for conventional photomultiplier tubes (PMTs). However, a significant disadvantage of SiPMs is crosstalk (CT), wherein photons propagate through other pixels, resulting in secondary avalanches. CT can be categorized into internal crosstalk and external crosstalk based on whether the secondary avalanche occurs within the same SiPM or a different one. Numerous methods exist for quantitatively estimating the percentage of internal crosstalk (iCT). However, external crosstalk (eCT) has not been extensively studied. This article presents a novel measurement method for the probability of emitting an external crosstalk photon during a single pixel avalanche, using a setup involving two identical SiPMs facing each other, and without the need for complex optical designs. The entire apparatus is enclosed within a stainless steel chamber, functioning as a light-tight enclosure, and maintained at liquid nitrogen temperature. The experimental setup incorporates two Sensl J-60035 SiPM chips along with two 0.5-inch Hamamatsu Photonics (HPK) VUV4 S13370-6050CN SiPM arrays. The findings show a linear relationship between the probability of emitting an external crosstalk photon and the SiPM overvoltage for both SiPM samples. Surprisingly, this novel measurement method also rovides measurements of the SiPM photon detection efficiency (PDE) for eCT photons at low temperature.

physics.ins-det↗

arXiv2024

Stoner instabilities and Ising excitonic states in twisted transition metal dichalcogenides

Moiré transition metal dichalcogenide (TMD) systems provide a tunable platform for studying electron-correlation driven quantum phases. Such phases have so far been found at rational fillings of the moiré superlattice, and it is believed that lattice commensurability plays a key role in their stability. In this work, we show via magnetotransport measurements on twisted WSe2 that new correlated electronic phases can exist away from commensurability. The first phase is an antiferromagnetic metal that is driven by proximity to the van Hove singularity. The second is a re-entrant magnetic field-driven insulator. This insulator is formed from a small and equal density of electrons and holes with opposite spin projections - an Ising excitonic insulator.

cond-mat.str-el↗

arXiv2024

Attainability of the best constant of Hardy-Sobolev inequality with full boundary singularities

We consider a type of Hardy-Sobolev inequality, whose weight function is singular on the whole domain boundary. We are concerned with the attainability of the best constant of such inequality. In dimension two, we link the inequality to a conformally invariant one using the conformal radius of the domain. The best constant of such inequality on a smooth bounded domain is achieved if and only if the domain is non-convex. In higher dimensions, the best constant is achieved if the domain has negative mean curvature somewhere. If the mean curvature vanishes but is non-umbilic somewhere, we also establish the attainability for some special cases. In the other direction, we also show the best constant is not achieved if the domain is sufficiently close to a ball in $C^2$ sense.

math.AP↗

arXiv2024

Information Leakage from Embedding in Large Language Models

The widespread adoption of large language models (LLMs) has raised concerns regarding data privacy. This study aims to investigate the potential for privacy invasion through input reconstruction attacks, in which a malicious model provider could potentially recover user inputs from embeddings. We first propose two base methods to reconstruct original texts from a model's hidden states. We find that these two methods are effective in attacking the embeddings from shallow layers, but their effectiveness decreases when attacking embeddings from deeper layers. To address this issue, we then present Embed Parrot, a Transformer-based method, to reconstruct input from embeddings in deep layers. Our analysis reveals that Embed Parrot effectively reconstructs original inputs from the hidden states of ChatGLM-6B and Llama2-7B, showcasing stable performance across various token lengths and data distributions. To mitigate the risk of privacy breaches, we introduce a defense mechanism to deter exploitation of the embedding reconstruction process. Our findings emphasize the importance of safeguarding user privacy in distributed learning systems and contribute valuable insights to enhance the security protocols within such environments.

cs.LG↗

arXiv2024

Bridging the Gap Between Domain-specific Frameworks and Multiple Hardware Devices

The rapid development of domain-specific frameworks has presented us with a significant challenge: The current approach of implementing solutions on a case-by-case basis incurs a theoretical complexity of O(M*N), thereby increasing the cost of porting applications to different hardware platforms. To address these challenges, we propose a systematic methodology that effectively bridges the gap between domain-specific frameworks and multiple hardware devices, reducing porting complexity to O(M+N). The approach utilizes multi-layer abstractions. Different domain-specific abstractions are employed to represent applications from various domains. These abstractions are then transformed into a unified abstraction, which is subsequently translated into combinations of primitive operators. Finally, these operators are mapped to multiple hardware platforms. The implemented unified framework supports deep learning, classical machine learning, and data analysis across X86, ARM, RISC-V, IoT devices, and GPU. It outperforms existing solutions like scikit-learn, hummingbird, Spark, and pandas, achieving impressive speedups: 1.1x to 3.83x on X86 servers, 1.06x to 4.33x on ARM IoT devices, 1.25x to 3.72x on RISC-V IoT devices, and 1.93x on GPU. The source code is available at https://github.com/BenchCouncil/bridger.git.

cs.SE↗

arXiv2024

Strongly coupled magneto-exciton condensates in large-angle twisted double bilayer graphene

Excitons, the bosonic quasiparticle emerging from Coulomb interaction between electrons and holes, will undergo a Bose-Einstein condensation(BEC) and transition into a superfluid state with global phase coherence at low temperatures. An important platform to study such excitonic physics is built on double-layer quantum wells or recent two-dimensional material heterostructures, where two parallel planes of electrons and holes are separated by a thin insulating layer. Lowering this separation distance ($d$) enhances the interlayer Coulomb interaction thereby strengthens the exciton binding energy. However, an exceedingly small $d$ will lead to the undesired interlayer tunneling, which results the annihilation of excitons. Here, we report the observation of a sequences of robust exciton condensates(ECs) in double bilayer graphenes twisted to $\sim 10^\circ$ with no insulating mid-layer. The large momentum mismatch between the two graphene layers well suppress the interlayer tunneling, allowing us to reach the separation lower limit $\sim$ 0.334 nm and investigate ECs in the extreme coupling regime. Carrying out transport measurements on the bulk and edge of the devices, we find incompressible states corresponding to ECs when both layers are half-filled in the $N=0$ and $N=1$ Landau levels (LLs). The comparison between these ECs and theoretical calculations suggest that the low-energy charged excitation of ECs can be meron-antimeron or particle-hole pair, which relies on both LL index and carrier type. Our results establish large-angle twisted bilayers as an experimental platform with extreme coupling strength for studying quantum bosonic phase and its low-energy excitations.

cond-mat.mes-hall↗

arXiv2024

A thermodynamic and analytical description on the quantitative phase-field model with enhanced interface diffusivity

Based on the idea of maintaining physical diffuse interface kinetics, enhancing interfacial diffusivity has recently provided a new direction for quantitative phase-field simulation at microstructural length and time scale. Establishing a general relationship between interface diffusivity and width is vital to facilitate the practical application. However, it is still limited by time-consuming numerical corrections, and its relationship with non-dilute thermodynamic properties still needs to be revealed. In this study, we present a new thermodynamic and analytical method for determining interfacial diffusivity enhancement. Unlike previous numerical corrections of partition coefficients and interface temperature, this new method aims to keep several thermodynamic quantities unchanged after enlarging the interface width. These essential quantities are theoretically proven to be diffusion potential jump across the diffuse interface and free energy dissipation by trans-interface diffusion. Since no dilute approximation has been employed in model derivation, the present method is available for binary alloys with arbitrary thermodynamic properties and can be easily extended to describe multicomponent systems. Therefore, the present method is expected to advance the recent quantitative phase-field framework and facilitate its practical applications.

cond-mat.mtrl-sci↗

arXiv2024

Accelerating Hybrid Federated Learning Convergence under Partial Participation

Over the past few years, Federated Learning (FL) has become a popular distributed machine learning paradigm. FL involves a group of clients with decentralized data who collaborate to learn a common model under the coordination of a centralized server, with the goal of protecting clients' privacy by ensuring that local datasets never leave the clients and that the server only performs model aggregation. However, in realistic scenarios, the server may be able to collect a small amount of data that approximately mimics the population distribution and has stronger computational ability to perform the learning process. To address this, we focus on the hybrid FL framework in this paper. While previous hybrid FL work has shown that the alternative training of clients and server can increase convergence speed, it has focused on the scenario where clients fully participate and ignores the negative effect of partial participation. In this paper, we provide theoretical analysis of hybrid FL under clients' partial participation to validate that partial participation is the key constraint on convergence speed. We then propose a new algorithm called FedCLG, which investigates the two-fold role of the server in hybrid FL. Firstly, the server needs to process the training steps using its small amount of local datasets. Secondly, the server's calculated gradient needs to guide the participated clients' training and the server's aggregation. We validate our theoretical findings through numerical experiments, which show that our proposed method FedCLG outperforms state-of-the-art methods.

cs.DC↗

arXiv2024

Taylor Videos for Action Recognition

Effectively extracting motions from video is a critical and long-standing problem for action recognition. This problem is very challenging because motions (i) do not have an explicit form, (ii) have various concepts such as displacement, velocity, and acceleration, and (iii) often contain noise caused by unstable pixels. Addressing these challenges, we propose the Taylor video, a new video format that highlights the dominate motions (e.g., a waving hand) in each of its frames named the Taylor frame. Taylor video is named after Taylor series, which approximates a function at a given point using important terms. In the scenario of videos, we define an implicit motion-extraction function which aims to extract motions from video temporal block. In this block, using the frames, the difference frames, and higher-order difference frames, we perform Taylor expansion to approximate this function at the starting frame. We show the summation of the higher-order terms in the Taylor series gives us dominant motion patterns, where static objects, small and unstable motions are removed. Experimentally we show that Taylor videos are effective inputs to popular architectures including 2D CNNs, 3D CNNs, and transformers. When used individually, Taylor videos yield competitive action recognition accuracy compared to RGB videos and optical flow. When fused with RGB or optical flow videos, further accuracy improvement is achieved. Additionally, we apply Taylor video computation to human skeleton sequences, resulting in Taylor skeleton sequences that outperform the use of original skeletons for skeleton-based action recognition.

cs.CV↗

arXiv2024

Ditto: Quantization-aware Secure Inference of Transformers upon MPC

Due to the rising privacy concerns on sensitive client data and trained models like Transformers, secure multi-party computation (MPC) techniques are employed to enable secure inference despite attendant overhead. Existing works attempt to reduce the overhead using more MPC-friendly non-linear function approximations. However, the integration of quantization widely used in plaintext inference into the MPC domain remains unclear. To bridge this gap, we propose the framework named Ditto to enable more efficient quantization-aware secure Transformer inference. Concretely, we first incorporate an MPC-friendly quantization into Transformer inference and employ a quantization-aware distillation procedure to maintain the model utility. Then, we propose novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference. This approach significantly decreases both computation and communication overhead, leading to improvements in overall efficiency. We conduct extensive experiments on Bert and GPT2 models to evaluate the performance of Ditto. The results demonstrate that Ditto is about $3.14\sim 4.40\times$ faster than MPCFormer (ICLR 2023) and $1.44\sim 2.35\times$ faster than the state-of-the-art work PUMA with negligible utility degradation.

cs.CR↗

arXiv2024

In-context Autoencoder for Context Compression in a Large Language Model

We propose the In-context Autoencoder (ICAE), leveraging the power of a large language model (LLM) to compress a long context into short compact memory slots that can be directly conditioned on by the LLM for various purposes. ICAE is first pretrained using both autoencoding and language modeling objectives on massive text data, enabling it to generate memory slots that accurately and comprehensively represent the original context. Then, it is fine-tuned on instruction data for producing desirable responses to various prompts. Experiments demonstrate that our lightweight ICAE, introducing about 1% additional parameters, effectively achieves $4\times$ context compression based on Llama, offering advantages in both improved latency and GPU memory cost during inference, and showing an interesting insight in memorization as well as potential for scalability. These promising results imply a novel perspective on the connection between working memory in cognitive science and representation learning in LLMs, revealing ICAE's significant implications in addressing the long context problem and suggesting further research in LLM context management. Our data, code and models are available at https://github.com/getao/icae.

cs.CL↗