arXiv subjects

Chao Zhang

Publications and source records attributed to Chao Zhang.

At least 505 records · Page 28Linked to original sources

arXiv2024

DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Driving

In autonomous driving, deep models have shown remarkable performance across various visual perception tasks with the demand of high-quality and huge-diversity training datasets. Such datasets are expected to cover various driving scenarios with adverse weather, lighting conditions and diverse moving objects. However, manually collecting these data presents huge challenges and expensive cost. With the rapid development of large generative models, we propose DriveDiTFit, a novel method for efficiently generating autonomous Driving data by Fine-tuning pre-trained Diffusion Transformers (DiTs). Specifically, DriveDiTFit utilizes a gap-driven modulation technique to carefully select and efficiently fine-tune a few parameters in DiTs according to the discrepancy between the pre-trained source data and the target driving data. Additionally, DriveDiTFit develops an effective weather and lighting condition embedding module to ensure diversity in the generated data, which is initialized by a nearest-semantic-similarity initialization approach. Through progressive tuning scheme to refined the process of detail generation in early diffusion process and enlarging the weights corresponding to small objects in training loss, DriveDiTFit ensures high-quality generation of small moving objects in the generated data. Extensive experiments conducted on driving datasets confirm that our method could efficiently produce diverse real driving data. The source codes will be available at https://github.com/TtuHamg/DriveDiTFit.

cs.CV↗

arXiv2024

SA-DS: A Dataset for Large Language Model-Driven AI Accelerator Design Generation

In the ever-evolving landscape of Deep Neural Networks (DNN) hardware acceleration, unlocking the true potential of systolic array accelerators has long been hindered by the daunting challenges of expertise and time investment. Large Language Models (LLMs) offer a promising solution for automating code generation which is key to unlocking unprecedented efficiency and performance in various domains, including hardware descriptive code. The generative power of LLMs can enable the effective utilization of preexisting designs and dedicated hardware generators. However, the successful application of LLMs to hardware accelerator design is contingent upon the availability of specialized datasets tailored for this purpose. To bridge this gap, we introduce the Systolic Array-based Accelerator Data Set (SA-DS). SA-DS comprises a diverse collection of spatial array designs following the standardized Berkeley's Gemmini accelerator generator template, enabling design reuse, adaptation, and customization. SA-DS is intended to spark LLM-centered research on DNN hardware accelerator architecture. We envision that SA-DS provides a framework that will shape the course of DNN hardware acceleration research for generations to come. SA-DS is open-sourced under the permissive MIT license at https://github.com/ACADLab/SA-DS.git}{https://github.com/ACADLab/SA-DS.

cs.AR↗

arXiv2024

Elliptic anisotropy of open-charm hadrons from parton scatterings in p--Pb collisions at energies available at the CERN Large Hadron Collider

The elliptic azimuthal anisotropy coefficient ($v_{2}$) of open-charm hadron at midrapidity ($|η<1|$) was studied in p--Pb collisions at $\sqrt{s_{\mathrm{NN}}}=$ 8.16 TeV using a multi-phase transport (AMPT) model included an additional charm quark--antiquark pair production trigger. The model provides a simultaneously description of the measured $p_{\mathrm{T}}$ spectrum and $v_{2}$ of $D^{0}$ meson, as well as the $v_{2}$ of light flavor meson $K_{s}^{0}$. We found that the $D^{0}$ and $K_{s}^{0}$ $v_{2}$ are both significantly affected by different parton scatterings among charm and light quarks. In addition, the predictions for the $v_{2}$ of other charm hadrons including $D^{+}$, $D_{s}^{+}$ and $Λ_{c}^{+}$ in p--Pb collisions are given for the first time. The $v_{2}$ of open-charm hadron reasonably follows the number-of-constituent-quark (NCQ) scaling up to 2.5 GeV, strongly indicating the importance of partonic degrees of freedom for the collectivity of heavy flavors in high-multiplicity p--Pb collisions. These findings may hint the formation of deconfined matter in small collision systems, and provide referential value for future measurements of azimuthal anisotropy at the LHC energies.

hep-ph↗

arXiv2024

Goal-Oriented State Information Compression for Linear Dynamical System Control

In this paper, we consider controlled linear dynamical systems in which the controller has only access to a compressed version of the system state. The technical problem we investigate is that of allocating compression resources over time such that the control performance degradation induced by compression is minimized. This can be formulated as an optimization problem to find the optimal resource allocation policy. Under mild assumptions, this optimization problem can be proved to have the same well-known structure as in [1], allowing the optimal resource allocation policy to be determined in closed-form. The obtained insights behind the optimal policy provide clear guidelines on the issue of "when to communicate" and "how to communicate" in dynamical systems with restricted communication resources. The obtained simulation results confirm the efficiency of the proposed allocation policy and illustrate the gain over the widely used uniform rate allocation policy.

eess.SY↗

arXiv2024

Linear instability of hairy black holes in Horndeski theory

Horndeski theory constitutes the most general model of scalar-tensor theories. It has attracted much attention in recent years in relation with black holes, celestial dynamics, stability analysis, etc. It is important to note that, for certain subclasses of Horndeski models, one can obtain analytic solutions for the background fields. This facilitates the investigation of the corresponding stability problems in detail. In particular, we aim to determine the constraints to the model or theory under which the stability conditions can be satisfied. In this study, we focused on a subclass of Horndeski theory and a set of analytic background solutions. In addition, the odd-parity gravitational perturbation and 2nd-order Lagrangian were investigated. Through careful analysis, the instability was identified within the neighborhood of the event horizon. This allows exclusion of a specific geometry for the model. Such an instability is implanted in the structure of the corresponding Lagrangian and is not erased by simply adding numerical constraints on the coupling parameters. As a starting point of our research, the current study provides insights for further exploration of the Horndeski theory.

gr-qc↗

arXiv2024

Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Models

The early detection of suicide risk is important since it enables the intervention to prevent potential suicide attempts. This paper studies the automatic detection of suicide risk based on spontaneous speech from adolescents, and collects a Mandarin dataset with 15 hours of suicide speech from more than a thousand adolescents aged from ten to eighteen for our experiments. To leverage the diverse acoustic and linguistic features embedded in spontaneous speech, both the Whisper speech model and textual large language models (LLMs) are used for suicide risk detection. Both all-parameter finetuning and parameter-efficient finetuning approaches are used to adapt the pre-trained models for suicide risk detection, and multiple audio-text fusion approaches are evaluated to combine the representations of Whisper and the LLM. The proposed system achieves a detection accuracy of 0.807 and an F1-score of 0.846 on the test set with 119 subjects, indicating promising potential for real suicide risk detection applications.

cs.CL↗

arXiv2024

PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer

Incremental Learning (IL) aims to learn deep models on sequential tasks continually, where each new task includes a batch of new classes and deep models have no access to task-ID information at the inference time. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples (rehearsal-free) and with a memory constraint (memory-constrained): Prompt-extending and Prompt-fixed methods. However, prompt-extending methods need a large memory buffer to maintain an ever-expanding prompt pool and meet an extra challenging prompt selection problem. Prompt-fixed methods only learn a single set of prompts on one of the incremental tasks and can not handle all the incremental tasks effectively. To achieve a good balance between the memory cost and the performance on all the tasks, we propose a Parameter-Efficient Cross-Task Prompt (PECTP) framework with Prompt Retention Module (PRM) and classifier Head Retention Module (HRM). To make the final learned prompts effective on all incremental tasks, PRM constrains the evolution of cross-task prompts' parameters from Outer Prompt Granularity and Inner Prompt Granularity. Besides, we employ HRM to inherit old knowledge in the previously learned classifier heads to facilitate the cross-task prompts' generalization ability. Extensive experiments show the effectiveness of our method. The source codes will be available at \url{https://github.com/RAIAN08/PECTP}.

cs.CV↗

arXiv2024

Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization

In this paper, we propose reverse inference optimization (RIO), a simple and effective method designed to enhance the robustness of autoregressive-model-based zero-shot text-to-speech (TTS) systems using reinforcement learning from human feedback (RLHF). To assess the quality of speech produced by the TTS system without human annotations, RIO introduces a novel concept termed as reverse inference based on the Bayesian principle, which suggests that a high-quality generated speech should be able to be used as a prompt for subsequent generation using the same TTS model. By leveraging reverse inference as the standard to select exemplars used in RLHF from the speech samples generated by the TTS system itself, RIO steers the subsequent optimization towards a direction of enhancing the TTS robustness. The RIO framework, comprising sampling, automatic annotating, and learning, obviates the need for a reward model or pairwise preference data, and significantly improves the stability of zero-shot TTS performance by reducing the discrepancies between training and inference conditions. Our experimental results verify that RIO can effectively improve both subjective and objective metrics, including mean opinion scores, word error rates, and speaker similarity. Remarkably, RIO can also diminish the incidence of bad outputs to nearly zero percent, rivalling the robustness when using ground-truth speech as the prompt.

cs.CL↗

arXiv2024

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). In this work, we propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-turbo-2024-0409, and ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks. Specifically, our Llama3-RankRAG significantly outperforms Llama3-ChatQA-1.5 and GPT-4 models on nine knowledge-intensive benchmarks. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.

cs.CL↗

arXiv2024

Learning Graph Structures and Uncertainty for Accurate and Calibrated Time-series Forecasting

Multi-variate time series forecasting is an important problem with a wide range of applications. Recent works model the relations between time-series as graphs and have shown that propagating information over the relation graph can improve time series forecasting. However, in many cases, relational information is not available or is noisy and reliable. Moreover, most works ignore the underlying uncertainty of time-series both for structure learning and deriving the forecasts resulting in the structure not capturing the uncertainty resulting in forecast distributions with poor uncertainty estimates. We tackle this challenge and introduce STOIC, that leverages stochastic correlations between time-series to learn underlying structure between time-series and to provide well-calibrated and accurate forecasts. Over a wide-range of benchmark datasets STOIC provides around 16% more accurate and 14% better-calibrated forecasts. STOIC also shows better adaptation to noise in data during inference and captures important and useful relational information in various benchmarks.

cs.LG↗

arXiv2024

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

Inference with modern Large Language Models (LLMs) is expensive and time-consuming, and speculative sampling has proven to be an effective solution. Most speculative sampling methods such as EAGLE use a static draft tree, implicitly assuming that the acceptance rate of draft tokens depends only on their position. Interestingly, we found that the acceptance rate of draft tokens is also context-dependent. In this paper, building upon EAGLE, we propose EAGLE-2, which introduces a new technique of context-aware dynamic draft tree into drafting modeling. This improvement leverages the fact that the draft model of EAGLE is well-calibrated: the confidence scores from the draft model approximate acceptance rates with small errors. We conducted extensive evaluations on three series of LLMs and six tasks, with EAGLE-2 achieving speedup ratios 3.05x-4.26x, which is 20%-40% faster than EAGLE-1. EAGLE-2 also ensures that the distribution of the generated text remains unchanged, making it a lossless acceleration algorithm.

cs.CL↗

arXiv2024

Even- and odd-parity stabilities of black holes in Einstein-Aether gravity

In Einstein-Aether theories with a timelike unit vector field, we study the linear stability of static and spherically symmetric black holes against both even- and odd-parity perturbations. For this purpose, we formulate a gauge-invariant black hole perturbation theory in the background Aether-orthogonal frame where the spacelike property of hypersurfaces orthogonal to the timelike Aether field is always maintained even inside the metric horizon. Using a short-wavelength approximation with large radial and angular momenta, we show that, in general, there are three dynamical degrees of freedom arising from the even-parity sector besides two propagating degrees of freedom present in the odd-parity sector. The propagation speeds of even-parity perturbations and their no-ghost conditions coincide with those of tensor, vector, and scalar perturbations on the Minkowski background, while the odd sector contains tensor and vector modes with the same propagation speeds as those in the even-parity sector (and hence as those on the Minkowski background). Thus, the consistent study of black hole perturbations in the Aether-orthogonal frame on static and spherically symmetric backgrounds does not add new small-scale stability conditions to those known for the Minkowski background in the literature.

gr-qc↗

arXiv2024

SAML: Speaker Adaptive Mixture of LoRA Experts for End-to-End ASR

Mixture-of-experts (MoE) models have achieved excellent results in many tasks. However, conventional MoE models are often very large, making them challenging to deploy on resource-constrained edge devices. In this paper, we propose a novel speaker adaptive mixture of LoRA experts (SAML) approach, which uses low-rank adaptation (LoRA) modules as experts to reduce the number of trainable parameters in MoE. Specifically, SAML is applied to the quantised and personalised end-to-end automatic speech recognition models, which combines test-time speaker adaptation to improve the performance of heavily compressed models in speaker-specific scenarios. Experiments have been performed on the LibriSpeech and the TED-LIUM 3 corpora. Remarkably, with a 7x reduction in model size, 29.1% and 31.1% relative word error rate reductions were achieved on the quantised Whisper model and Conformer-based attention-based encoder-decoder ASR model respectively, comparing to the original full precision models.

cs.SD↗

arXiv2024

Unified Framework for Calculating Convex Roof Resource Measures

Quantum resource theories (QRTs) provide a comprehensive and practical framework for the analysis of diverse quantum phenomena. A fundamental task within QRTs is the quantification of resources inherent in a given quantum state. In this letter, we introduce a unified computational framework for a class of widely utilized quantum resource measures, derived from convex roof extensions. We establish that the computation of these convex roof resource measures can be reformulated as an optimization problem over a Stiefel manifold, which can be further unconstrained through polar projection. Compared to existing methods employing semi-definite programming (SDP), gradient-based techniques or seesaw strategy, our approach not only demonstrates superior computational efficiency but also maintains applicability across various scenarios within a streamlined workflow. We substantiate the efficacy of our method by applying it to several key quantum resources, including entanglement, coherence, and magic states. Moreover, our methodology can be readily extended to other convex roof quantities beyond the domain of resource theories, suggesting broad applicability in the realm of quantum information theory.

quant-ph↗

arXiv2024

Existence and uniqueness of weak solutions to a parabolic nonlocal 1-Laplacian equation

We consider a class of parabolic nonlocal $1$-Laplacian equation \begin{align*} u_t+(-Δ)^s_1u=f \quad \text{ in }Ω\times(0,T]. \end{align*} By employing the Rothe time-discretization method, we establish the existence and uniqueness of weak solutions to the equation above. In particular, different from the previous results on the local case, we infer that the weak solution maintains $\frac{1}{2}$-Hölder continuity in time.

math.AP↗

arXiv2024

video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models

Speech understanding as an element of the more generic video understanding using audio-visual large language models (av-LLMs) is a crucial yet understudied aspect. This paper proposes video-SALMONN, a single end-to-end av-LLM for video processing, which can understand not only visual frame sequences, audio events and music, but speech as well. To obtain fine-grained temporal information required by speech understanding, while keeping efficient for other video elements, this paper proposes a novel multi-resolution causal Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders and the backbone large language model. Moreover, dedicated training approaches including the diversity loss and the unpaired audio-visual mixed training scheme are proposed to avoid frames or modality dominance. On the introduced speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\% absolute accuracy improvements on the video-QA task and over 30\% absolute accuracy improvements on audio-visual QA tasks with human speech. In addition, video-SALMONN demonstrates remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other av-LLMs. Our training code and model checkpoints are available at \texttt{\url{https://github.com/bytedance/SALMONN/}}.

cs.CV↗

arXiv2024

Accelerating materials discovery for polymer solar cells: Data-driven insights enabled by natural language processing

We present a simulation of various active learning strategies for the discovery of polymer solar cell donor/acceptor pairs using data extracted from the literature spanning $\sim$20 years by a natural language processing pipeline. While data-driven methods have been well established to discover novel materials faster than Edisonian trial-and-error approaches, their benefits have not been quantified for material discovery problems that can take decades. Our approach demonstrates a potential reduction in discovery time by approximately 75 %, equivalent to a 15 year acceleration in material innovation. Our pipeline enables us to extract data from greater than 3300 papers which is $\sim$5 times larger and therefore more diverse than similar data sets reported by others. We also trained machine learning models to predict the power conversion efficiency and used our model to identify promising donor-acceptor combinations that are as yet unreported. We thus demonstrate a pipeline that goes from published literature to extracted material property data which in turn is used to obtain data-driven insights. Our insights include active learning strategies that can be used to train strong predictive models of material properties or be robust to the initial material system used. This work provides a valuable framework for data-driven research in materials science.

cond-mat.mtrl-sci↗

arXiv2024

LLMatDesign: Autonomous Materials Discovery with Large Language Models

Discovering new materials can have significant scientific and technological implications but remains a challenging problem today due to the enormity of the chemical space. Recent advances in machine learning have enabled data-driven methods to rapidly screen or generate promising materials, but these methods still depend heavily on very large quantities of training data and often lack the flexibility and chemical understanding often desired in materials discovery. We introduce LLMatDesign, a novel language-based framework for interpretable materials design powered by large language models (LLMs). LLMatDesign utilizes LLM agents to translate human instructions, apply modifications to materials, and evaluate outcomes using provided tools. By incorporating self-reflection on its previous decisions, LLMatDesign adapts rapidly to new tasks and conditions in a zero-shot manner. A systematic evaluation of LLMatDesign on several materials design tasks, in silico, validates LLMatDesign's effectiveness in developing new materials with user-defined target properties in the small data regime. Our framework demonstrates the remarkable potential of autonomous LLM-guided materials discovery in the computational setting and towards self-driving laboratories in the future.

cond-mat.mtrl-sci↗