arXiv subjects

Chao Zhang

Publications and source records attributed to Chao Zhang.

At least 415 records · Page 23Linked to original sources

arXiv2025

NoteLLM-2: Multimodal Large Representation Models for Recommendation

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.

cs.IR↗

arXiv2025

Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer

Despite the significant progress in controllable music generation and editing, challenges remain in the quality and length of generated music due to the use of Mel-spectrogram representations and UNet-based model structures. To address these limitations, we propose a novel approach using a Diffusion Transformer (DiT) augmented with an additional control branch using ControlNet. This allows for long-form and variable-length music generation and editing controlled by text and melody prompts. For more precise and fine-grained melody control, we introduce a novel top-$k$ constant-Q Transform representation as the melody prompt, reducing ambiguity compared to previous representations (e.g., chroma), particularly for music with multiple tracks or a wide range of pitch values. To effectively balance the control signals from text and melody prompts, we adopt a curriculum learning strategy that progressively masks the melody prompt, resulting in a more stable training process. Experiments have been performed on text-to-music generation and music-style transfer tasks using open-source instrumental recording data. The results demonstrate that by extending StableAudio, a pre-trained text-controlled DiT model, our approach enables superior melody-controlled editing while retaining good text-to-music generation performance. These results outperform a strong MusicGen baseline in terms of both text-based generation and melody preservation for editing. Audio examples can be found at https://stable-audio-control.github.io.

eess.AS↗

arXiv2025

SA-MLP: A Low-Power Multiplication-Free Deep Network for 3D Point Cloud Classification in Resource-Constrained Environments

Point cloud classification plays a crucial role in the processing and analysis of data from 3D sensors such as LiDAR, which are commonly used in applications like autonomous vehicles, robotics, and environmental monitoring. However, traditional neural networks, which rely heavily on multiplication operations, often face challenges in terms of high computational costs and energy consumption. This study presents a novel family of efficient MLP-based architectures designed to improve the computational efficiency of point cloud classification tasks in sensor systems. The baseline model, Mul-MLP, utilizes conventional multiplication operations, while Add-MLP and Shift-MLP replace multiplications with addition and shift operations, respectively. These replacements leverage more sensor-friendly operations that can significantly reduce computational overhead, making them particularly suitable for resource-constrained sensor platforms. To further enhance performance, we propose SA-MLP, a hybrid architecture that alternates between shift and adder layers, preserving the network depth while optimizing computational efficiency. Unlike previous approaches such as ShiftAddNet, which increase the layer count and limit representational capacity by freezing shift weights, SA-MLP fully exploits the complementary advantages of shift and adder layers by employing distinct learning rates and optimizers. Experimental results show that Add-MLP and Shift-MLP achieve competitive performance compared to Mul-MLP, while SA-MLP surpasses the baseline, delivering results comparable to state-of-the-art MLP models in terms of both classification accuracy and computational efficiency. This work offers a promising, energy-efficient solution for sensor-driven applications requiring real-time point cloud classification, particularly in environments with limited computational resources.

cs.CV↗

arXiv2025

Multi-episodes of remote brightenings driven by a coronal EUV jet on the Sun

Remote brightening (RB) is compact brightening at footpoints of magnetic loops, which are remotely-connecting to and confining an eruption in the solar atmosphere. Here, we report on observations of an RB resulting from an EUV jet with a speed of about 90\,km\,s$^{-1}$. The loops connecting the RB and the jet have an apparent length of about 59\,Mm. Intriguingly, the RB exhibits at least two episodes of brightenings, as characterised by two peaks in its lightcurve. The energies, which sustain the first and second peaks of the RB, are $6.3\times10^{26}$\,erg and $8.4\times10^{26}$\,erg, respectively, and take a significant proportion of the total energy. The first peak of the RB brightenings coincides with the jet's peak with a time delay of 12 seconds, while the second peak lags behind by 108 seconds. Besides the flows of the ejecta, we have identified two additional flows originating from the eruption site. One is relatively cool with a temperature of $log_{10}T/K=5.8-6.1$ and has a speed of about $275\pm15$\,km\,s$^{-1}$. The other is hot with a temperature of $log_{10}T/K=7.0-7.3$ and travels much faster with a speed of about 750$\pm$\,70\,km\,s$^{-1}$. We attribute the second peak of RB directly to this hot flow, which our numerical experiments suggest is the result of a slow shock wave. Considering the minimal time delay between the first peak of RB and the eruption, we infer this first episode is due to heating by nonthermal electrons. Our research demonstrates that the dynamics in an RB can offer vital insights into the nature of the corresponding eruption and help understand how the energy is distributed throughout the solar atmosphere.

astro-ph.SR↗

arXiv2025

On How Traffic Signals Impact the Fundamental Diagrams of Urban Roads

Being widely adopted by the transportation and planning practitioners, the fundamental diagram (FD) is the primary tool used to relate the key macroscopic traffic variables of speed, flow, and density. We empirically analyze the relation between vehicular space-mean speeds and flows given different signal settings and postulate a parsimonious parametric function form of the traditional FD where its function parameters are explicitly modeled as a function of the signal plan factors. We validate the proposed formulation using data from signalized urban road segments in Salt Lake City, Utah, USA. The proposed formulation builds our understanding of how changes to signal settings impact the FDs, and more generally the congestion patterns, of signalized urban segments.

cs.ET↗

arXiv2025

TextToucher: Fine-Grained Text-to-Touch Generation

Tactile sensation plays a crucial role in the development of multi-modal large models and embodied intelligence. To collect tactile data with minimal cost as possible, a series of studies have attempted to generate tactile images by vision-to-touch image translation. However, compared to text modality, visual modality-driven tactile generation cannot accurately depict human tactile sensation. In this work, we analyze the characteristics of tactile images in detail from two granularities: object-level (tactile texture, tactile shape), and sensor-level (gel status). We model these granularities of information through text descriptions and propose a fine-grained Text-to-Touch generation method (TextToucher) to generate high-quality tactile samples. Specifically, we introduce a multimodal large language model to build the text sentences about object-level tactile information and employ a set of learnable text prompts to represent the sensor-level tactile information. To better guide the tactile generation process with the built text information, we fuse the dual grains of text information and explore various dual-grain text conditioning methods within the diffusion transformer architecture. Furthermore, we propose a Contrastive Text-Touch Pre-training (CTTP) metric to precisely evaluate the quality of text-driven generated tactile data. Extensive experiments demonstrate the superiority of our TextToucher method. The source codes will be available at \url{https://github.com/TtuHamg/TextToucher}.

cs.CV↗

arXiv2025

Traffic Simulations: Multi-City Calibration of Metropolitan Highway Networks

This paper proposes an approach to perform travel demand calibration for high-resolution stochastic traffic simulators. It employs abundant travel times at the path-level, departing from the standard practice of resorting to scarce segment-level sensor counts. The proposed approach is shown to tackle high-dimensional instances in a sample-efficient way. For the first time, case studies on 6 metropolitan highway networks are carried out, considering a total of 54 calibration scenarios. This is the first work to show the ability of a calibration algorithm to systematically scale across networks. Compared to the state-of-the-art simultaneous perturbation stochastic approximation (SPSA) algorithm, the proposed approach enhances fit to field data by an average 43.5% with a maximum improvement of 80.0%, and does so within fewer simulation calls.

cs.ET↗

arXiv2025

Low-Latency Sliding Window Connectivity

Connectivity queries, which check whether vertices belong to the same connected component, are fundamental in graph computations. Sliding window connectivity processes these queries over sliding windows, facilitating real-time streaming graph analytics. However, existing methods struggle with low-latency processing due to the significant overhead of continuously updating index structures as edges are inserted and deleted. We introduce a novel approach that leverages spanning trees to efficiently process queries. The novelty of this method lies in its ability to maintain spanning trees efficiently as window updates occur. Notably, our approach completely eliminates the need for replacement edge searches, a traditional bottleneck in managing spanning trees during edge deletions. We also present several optimizations to maximize the potential of spanning-tree-based indexes. Our comprehensive experimental evaluation shows that index update latency in spanning trees can be reduced by up to $458\times$ while maintaining query performance, leading to an $8\times$ improvement in throughput. Our approach also significantly outperforms the state-of-the-art in both query processing and index updates. Additionally, our methods use significantly less memory and demonstrate consistent efficiency across various settings.

cs.DB↗

arXiv2025

Detecting Defective Wafers Via Modular Networks

The growing availability of sensors within semiconductor manufacturing processes makes it feasible to detect defective wafers with data-driven models. Without directly measuring the quality of semiconductor devices, they capture the modalities between diverse sensor readings and can be used to predict key quality indicators (KQI, \textit{e.g.}, roughness, resistance) to detect faulty products, significantly reducing the capital and human cost in maintaining physical metrology steps. Nevertheless, existing models pay little attention to the correlations among different processes for diverse wafer products and commonly struggle with generalizability issues. To enable generic fault detection, in this work, we propose a modular network (MN) trained using time series stage-wise datasets that embodies the structure of the manufacturing process. It decomposes KQI prediction as a combination of stage modules to simulate compositional semiconductor manufacturing, universally enhancing faulty wafer detection among different wafer types and manufacturing processes. Extensive experiments demonstrate the usefulness of our approach, and shed light on how the compositional design provides an interpretable interface for more practical applications.

cs.LG↗

arXiv2025

FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models

Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.

cs.CV↗

arXiv2024

Investigating Pure State Uniqueness in Tomography via Optimization

Quantum state tomography (QST) is crucial for understanding and characterizing quantum systems through measurement data. Traditional QST methods face scalability challenges, requiring $\mathcal{O}(d^2)$ measurements for a general $d$-dimensional state. This complexity can be substantially reduced to $\mathcal{O}(d)$ in pure state tomography, indicating that full measurements are unnecessary for pure states. In this paper, we investigate the conditions under which a given pure state can be uniquely determined by a subset of full measurements, focusing on the concepts of uniquely determined among pure states (UDP) and uniquely determined among all states (UDA). The UDP determination inherently involves non-convexity challenges, while the UDA determination, though convex, becomes computationally intensive for high-dimensional systems. To address these issues, we develop a unified framework based on the Augmented Lagrangian Method (ALM). Specifically, our theorem on the existence of low-rank solutions in QST allows us to reformulate the UDA problem with low-rank constraints, thereby reducing the number of variables involved. Our approach entails parameterizing quantum states and employing ALM to handle the constrained non-convex optimization tasks associated with UDP and low-rank UDA determinations. Numerical experiments conducted on qutrit systems and four-qubit symmetric states not only validate theoretical findings but also reveal the complete distribution of quantum states across three uniqueness categories: (A) UDA, (B) UDP but not UDA, and (C) neither UDP nor UDA. This work provides a practical approach for determining state uniqueness, advancing our understanding of quantum state reconstruction.

quant-ph↗

arXiv2024

Space-Air-Ground Integrated Networks: Their Channel Model and Performance Analysis

Given their extensive geographic coverage, low Earth orbit (LEO) satellites are envisioned to find their way into next-generation (6G) wireless communications. This paper explores space-air-ground integrated networks (SAGINs) leveraging LEOs to support terrestrial and non-terrestrial users. We first propose a practical satellite-ground channel model that incorporates five key aspects: 1) the small-scale fading characterized by the Shadowed-Rician distribution in terms of the Rician factor K, 2) the path loss effect of bending rays due to atmospheric refraction, 3) the molecular absorption modelled by the Beer-Lambert law, 4) the Doppler effects including the Earth's rotation, and 5) the impact of weather conditions according to the International Telecommunication Union Recommendations (ITU-R). Harnessing the proposed model, we analyze the long-term performance of the SAGIN considered. Explicitly, the closed-form expressions of both the outage probability and of the ergodic rates are derived. Additionally, the upper bounds of bit-error rates and of the Goodput are investigated. The numerical results yield the following insights: 1) The shadowing effect and the ratio between the line-of-sight and scattering components can be conveniently modeled by the factors of K and m in the proposed Shadowed-Rician small-scale fading model. 2) The atmospheric refraction has a modest effect on the path loss. 3) When calculating the transmission distance of waves, Earth's curvature and its geometric relationship with the satellites must be considered, particularly at small elevation angles. 3) High-frequency carriers suffer from substantial path loss, and 4) the Goodput metric is eminently suitable for characterizing the performance of different coding as well as modulation methods and of the estimation error of the Doppler effects.

cs.IT↗

arXiv2024

Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Models

Significant advancements in Large Multimodal Models (LMMs) have enabled them to tackle complex problems involving visual-mathematical reasoning. However, their ability to identify geometric elements remains underexplored. To address this gap, we introduce Tangram, a novel benchmark designed to evaluate the performance of LMMs on geometric element recognition. Tangram comprises 1,080 diverse geometric diagrams sourced from primary and secondary school exams, competitions, and textbooks, ranging from simple geometric shapes to complex combinations. Each diagram is paired with four questions, resulting in 4,320 visual-question-answer pairs. Unlike existing benchmarks that emphasize higher-level cognition and reasoning, Tangram focuses on understanding geometric elements, requiring models to perform a ``simple yet challenging" counting task. Systematic evaluation of 13 prominent LMMs, such as GPT-4o and Claude 3.5 Sonnet, reveals that these models face significant challenges even in seemingly straightforward tasks. The top-performing model achieves an accuracy of only 53.0%, highlighting a substantial gap compared to human performance. These findings underscore the limitations of current multimodal AI systems in handling basic perception tasks and serve to inspire the development of the next generation of expert-level multimodal foundational models. The data and code will be released soon.

cs.CV↗

arXiv2024

Geometric Deep Learning for Realized Covariance Matrix Forecasting

Traditional methods employed in matrix volatility forecasting often overlook the inherent Riemannian manifold structure of symmetric positive definite matrices, treating them as elements of Euclidean space, which can lead to suboptimal predictive performance. Moreover, they often struggle to handle high-dimensional matrices. In this paper, we propose a novel approach for forecasting realized covariance matrices of asset returns using a Riemannian-geometry-aware deep learning framework. In this way, we account for the geometric properties of the covariance matrices, including possible non-linear dynamics and efficient handling of high-dimensionality. Moreover, building upon a Fréchet sample mean of realized covariance matrices, we are able to extend the HAR model to the matrix-variate. We demonstrate the efficacy of our approach using daily realized covariance matrices for the 50 most capitalized companies in the S&P 500 index, showing that our method outperforms traditional approaches in terms of predictive accuracy.

q-fin.CP↗

arXiv2024

Probing the equation of state of neutron stars with captured primordial black holes

Gravitational waves (GWs) from primordial black holes (PBHs) inspiraling within neutron stars (NSs) -- should they exist -- are detectable by ground-based detectors and offer a unique insight into the internal structure of NSs. To provide accurate templates for GW searches, we solve Einstein's equations within NSs and calculate the orbital motion of the captured PBH by considering dynamical friction, accretion, and gravitational radiation. Equipped with precise GW waveforms for PBHs inspiraling inside NSs, we find that the Einstein Telescope can differentiate between various equations of state for NSs. As PBHs inspiral deeper into NSs, the GW frequency rises near the surface, then decreases to a constant value deeper within NSs. The distinctive characteristics of GW frequency serve as the smoking gun for GW signals emitted by PBHs inspiraling inside NSs and can be used to probe the nuclear matter in the crust and core of NSs.

gr-qc↗

arXiv2024

Automated Dynamic Image Analysis for Particle Size and Shape Classification in Three Dimensions

We introduce OCULAR, an innovative hardware and software solution for three-dimensional dynamic image analysis of fine particles. Current state-of-the art instruments for dynamic image analysis are largely limited to two-dimensional imaging. However, extensive literature has demonstrated that relying on a single two-dimensional projection for particle characterisation can lead to inaccuracies in many applications. Existing three-dimensional imaging technologies, such as computed tomography, laser scanning, and orthophotography, are limited to static objects. These methods are often not statistically representative and come with significant post-processing requirements, as well as the need for specialised imaging and computing resources. OCULAR addresses these challenges by providing a cost-effective solution for imaging continuous particle streams using a synchronised array of optical cameras. Particle shape characterisation is achieved through the reconstruction of their three-dimensional surfaces. This paper details the OCULAR methodology, evaluates its reproducibility, and compares its results against X-ray micro computed tomography, highlighting its potential for efficient and reliable particle analysis.

cs.CV↗

arXiv2024

Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment

Implicit assumptions and priors are often necessary in text-to-image generation tasks, especially when textual prompts lack sufficient context. However, these assumptions can sometimes reflect outdated concepts, inaccuracies, or societal bias embedded in the training data. We present Embedding-only Editing (Embedit), a method designed to efficiently adjust implict assumptions and priors in the model without affecting its interpretation of unrelated objects or overall performance. Given a "source" prompt (e.g., "rose") that elicits an implicit assumption (e.g., rose is red) and a "destination" prompt that specifies the desired attribute (e.g., "blue rose"), Embedit fine-tunes only the word token embedding (WTE) of the target object ("rose") to optimize the last hidden state of text encoder in Stable Diffusion, a SOTA text-to-image model. This targeted adjustment prevents unintended effects on other objects in the model's knowledge base, as the WTEs for unrelated objects and the model weights remain unchanged. Consequently, when a prompt does not contain the edited object, all representations, and the model outputs are identical to those of the original, unedited model. Our method is highly efficient, modifying only 768 parameters for Stable Diffusion 1.4 and 2048 for XL in a single edit, matching the WTE dimension of each respective model. This minimal scope, combined with rapid execution, makes Embedit highly practical for real-world applications. Additionally, changes are easily reversible by restoring the original WTE layers. Our experimental results demonstrate that Embedit consistently outperforms previous methods across various models, tasks, and editing scenarios (both single and sequential multiple edits), achieving at least a 6.01% improvement (from 87.17% to 93.18%).

cs.CV↗

arXiv2024

Deep Reinforcement Learning for Energy-Efficient on the Heterogeneous Computing Architecture

The growing demand for optimal and low-power energy consumption paradigms for IOT devices has garnered significant attention due to their cost-effectiveness, simplicity, and intelligibility. In this article, an AI hardware energy-efficient framework to achieve optimal energy savings in heterogeneous computing through appropriate power consumption management is proposed. The deep reinforcement learning framework is employed, utilising the Actor-Critic architecture to provide a simple and precise method for power saving. The results of the study demonstrate the proposed approach's suitability for different hardware configurations, achieving notable energy consumption control while adhering to strict performance requirements. The evaluation of the proposed power-saving framework shows that it is more stable, and has achieved more than 34.6% efficiency improvement, outperforming other methods by more than 16%.

eess.SP↗