arXiv subjects

Jie Ren

Publications and source records attributed to Jie Ren.

At least 91 records · Page 5Linked to original sources

arXiv2025

Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data

Retrieval-augmented generation (RAG) enhances the outputs of language models by integrating relevant information retrieved from external knowledge sources. However, when the retrieval process involves private data, RAG systems may face severe privacy risks, potentially leading to the leakage of sensitive information. To address this issue, we propose using synthetic data as a privacy-preserving alternative for the retrieval data. We propose SAGE, a novel two-stage synthetic data generation paradigm. In the stage-1, we employ an attribute-based extraction and generation approach to preserve key contextual information from the original data. In the stage-2, we further enhance the privacy properties of the synthetic data through an agent-based iterative refinement process. Extensive experiments demonstrate that using our synthetic data as the retrieval context achieves comparable performance to using the original data while substantially reducing privacy risks. Our work takes the first step towards investigating the possibility of generating high-utility and privacy-preserving synthetic data for RAG, opening up new opportunities for the safe application of RAG systems in various domains.

cs.CR↗

arXiv2025

Exploring and Evaluating Real-world CXL: Use Cases and System Adoption

Compute eXpress Link (CXL) is emerging as a promising memory interface technology. However, its performance characteristics remain largely unclear due to the limited availability of production hardware. Key questions include: What are the use cases for the CXL memory? What are the impacts of the CXL memory on application performance? How to use the CXL memory in combination with existing memory components? In this work, we study the performance of three genuine CXL memory-expansion cards from different vendors. We characterize the basic performance of the CXL memory, study how HPC applications and large language models (LLM) can benefit from the CXL memory, and study the interplay between memory tiering and page interleaving. We also propose a novel data object-level interleaving policy to match the interleaving policy with memory access patterns. Our findings reveal the challenges and opportunities of using the CXL memory.

cs.PF↗

arXiv2025

Sensitivity of three-dimensional boundary-layer stability to intrinsic uncertainties of fluid properties: a study on supercritical CO2

The intrinsic uncertainty of fluid properties, including the equation of state, viscosity, and thermal conductivity, on boundary layer stability has scarcely been addressed. When a fluid is operating in the vicinity of the Widom line (defined as the maximum of isobaric specific heat) in supercritical state, its properties exhibit highly non-ideal behavior, which is an ongoing research field leading to refined and more accurate fluid property databases. Upon crossing the Widom line, new mechanisms of flow instability emerge, feasibly leading to changes in dominating modes that yield turbulence. The present work investigates the sensitivity of three-dimensional boundary-layer modal instability to these intrinsic uncertainties in fluid properties. The uncertainty, regardless of its source and the fluid regimes, gives rise to distortions of all profiles that constitute the inputs of the stability operator. The effect of these distortions on flow stability is measured by sensitivity coefficients, which are formulated with the adjoint operator and validated against linear modal stability analysis. The results are presented for carbon dioxide at a representative supercritical pressure of about 80 bar. The sensitivity to different inputs of the stability operator across various thermodynamic regimes show an immense range of sensitivity amplitude. A balancing relationship between the density gradient and its perturbation leads to a quadratic effect across the Widom line, provoking significant sensitivity to distortions of the second derivative of the pressure with respect to the density, $\partial^2 p/\partial ρ^2$. From an application-oriented point of view, one important question is whether the correct baseflow profiles can be meaningfully analyzed by the simplified ideal-fluid model...

physics.flu-dyn↗

arXiv2025

Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking

Direct preference optimization (DPO) has shown success in aligning diffusion models with human preference. Previous approaches typically assume a consistent preference label between final generations and noisy samples at intermediate steps, and directly apply DPO to these noisy samples for fine-tuning. However, we theoretically identify inherent issues in this assumption and its impacts on the effectiveness of preference alignment. We first demonstrate the inherent issues from two perspectives: gradient direction and preference order, and then propose a Tailored Preference Optimization (TailorPO) framework for aligning diffusion models with human preference, underpinned by some theoretical insights. Our approach directly ranks intermediate noisy samples based on their step-wise reward, and effectively resolves the gradient direction issues through a simple yet efficient design. Additionally, we incorporate the gradient guidance of diffusion models into preference alignment to further enhance the optimization effectiveness. Experimental results demonstrate that our method significantly improves the model's ability to generate aesthetically pleasing and human-preferred images.

cs.LG↗

arXiv2025

ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training

Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49x.

cs.DC↗

arXiv2024

Stealthy Backdoor Attack via Confidence-driven Sampling

Backdoor attacks aim to surreptitiously insert malicious triggers into DNN models, granting unauthorized control during testing scenarios. Existing methods lack robustness against defense strategies and predominantly focus on enhancing trigger stealthiness while randomly selecting poisoned samples. Our research highlights the overlooked drawbacks of random sampling, which make that attack detectable and defensible. The core idea of this paper is to strategically poison samples near the model's decision boundary and increase defense difficulty. We introduce a straightforward yet highly effective sampling methodology that leverages confidence scores. Specifically, it selects samples with lower confidence scores, significantly increasing the challenge for defenders in identifying and countering these attacks. Importantly, our method operates independently of existing trigger designs, providing versatility and compatibility with various backdoor attack techniques. We substantiate the effectiveness of our approach through a comprehensive set of empirical experiments, demonstrating its potential to significantly enhance resilience against backdoor attacks in DNNs.

cs.CR↗

arXiv2024

Exploring quantum criticality and ergodicity-breaking dynamics in spin-1 Kitaev chains via single-ion anisotropies

We investigate topological gauge-theory terms and quantum criticality in a spin-1 Kitaev chain with general single-ion anisotropies (SIAs). The ground-state phase diagram, including the Kitaev spin liquid (KSL) and gapless dimer phases, is determined by the infinite time evolving block decimation (iTEBD) method. A quantum phase transition between the KSL and dimer phases occurs by varying uniaxial SIA, analogous to the confinement-deconfinement transition in the lattice Schwinger model with a topological $θ$ angle of $π$. Introducing rhombic SIA shifts this angle from $π$, resulting in $y$- and $x$-ferroquadrupole phases. The transition between these phases can occur through a crossover in the KSL phase or a genuine phase transition along a deconfined line. We map the spin-1 Hamiltonian to an effective spin-1/2 PXP Hamiltonian, with uniaxial SIA corresponding to uniform detuning and rhombic SIA to staggered detuning. We explore the hierarchical fragmentation of the Hilbert space, revealing that quantum many-body scars (QMBSs) emerge under weak uniform detuning, while slow dynamics under large staggered detuning is accurately captured by a second-order effective Hamiltonian via the Schrieffer-Wolff transformation. Our work establishes a framework for simulating topological $θ$ angles and ergodicity-breaking dynamics, bridging higher-spin generalizations of scarred models with lattice gauge theories, potentially realizable using state-of-the-art cold-atom quantum simulators.

cond-mat.str-el↗

arXiv2024

Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models

Large Language Models (LLMs) and Vision-Language Models (VLMs) have made significant advancements in a wide range of natural language processing and vision-language tasks. Access to large web-scale datasets has been a key factor in their success. However, concerns have been raised about the unauthorized use of copyrighted materials and potential copyright infringement. Existing methods, such as sample-level Membership Inference Attacks (MIA) and distribution-based dataset inference, distinguish member data (data used for training) and non-member data by leveraging the common observation that models tend to memorize and show greater confidence in member data. Nevertheless, these methods face challenges when applied to LLMs and VLMs, such as the requirement for ground-truth member data or non-member data that shares the same distribution as the test data. In this paper, we propose a novel dataset-level membership inference method based on Self-Comparison. We find that a member prefix followed by a non-member suffix (paraphrased from a member suffix) can further trigger the model's memorization on training data. Instead of directly comparing member and non-member data, we introduce paraphrasing to the second half of the sequence and evaluate how the likelihood changes before and after paraphrasing. Unlike prior approaches, our method does not require access to ground-truth member data or non-member data in identical distribution, making it more practical. Extensive experiments demonstrate that our proposed method outperforms traditional MIA and dataset inference techniques across various datasets and models, including including public models, fine-tuned models, and API-based commercial models.

cs.LG↗

arXiv2024

Accelerating Mixed-Precision Out-of-Core Cholesky Factorization with Static Task Scheduling

This paper explores the performance optimization of out-of-core (OOC) Cholesky factorization on shared-memory systems equipped with multiple GPUs. We employ fine-grained computational tasks to expose concurrency while creating opportunities to overlap data movement asynchronously with computations, especially when dealing with matrices that cannot fit on the GPU memory. We leverage the directed acyclic graph of the task-based Cholesky factorization and map it onto a static scheduler that promotes data reuse while supporting strategies for reducing data movement with the CPU host when the GPU memory is exhausted. The CPU-GPU interconnect may become the main performance bottleneck as the gap between the GPU execution rate and the traditional PCIe bandwidth continues to widen. While the surface-to-volume effect of compute-bound kernels partially mitigates the overhead of data motion, deploying mixed-precision (MxP) computations exacerbates the throughput discrepancy. Using static task scheduling, we evaluate the performance capabilities of the new ultra-fast NVIDIA chip interconnect technology, codenamed NVLink-C2C, that constitutes the backbone of the NVIDIA Grace Hopper Superchip (GH200), against a new four-precision (FP64/FP32/FP16/FP8) left-looking Cholesky factorization. We report the performance results of a benchmarking campaign on various NVIDIA GPU generations and interconnects. We highlight 20% performance superiority against cuSOLVER on a single GH200 with FP64 while hiding the cost of OOC task-based Cholesky factorization, and we scale almost linearly on four GH200 superships. With MxP enabled, our statically scheduled four-precision tile-based Cholesky factorization scores a 3X performance speedup against its FP64-only counterpart, delivering application-worthy FP64 accuracy when modeling a large-scale geospatial statistical application.

cs.DC↗

arXiv2024

Superdiffusive transport on lattices with nodal impurities

We show that 1D lattice models exhibit superdiffusive transport in the presence of random "nodal impurities" in the absence of interaction. Here a nodal impurity is defined as a localized state, the wave function of which has zeros (nodes) in momentum space. The dynamics exponent $z$, a defining quantity for transport behaviors, is computed to establish this result. To be specific, in a disordered system having only nodal impurities, the dynamical exponent $z=4n/(4n-1)$ where $n$ is the order of the node. If the system has time reversal, the nodes appear in pairs and the dynamical exponent can be enhanced to $z=8n/(8n-1)$. As $1<z<2$, both cases indicate superdiffusive transport.

cond-mat.mes-hall↗

arXiv2024

Toward Capturing Genetic Epistasis From Multivariate Genome-Wide Association Studies Using Mixed-Precision Kernel Ridge Regression

We exploit the widening margin in tensor-core performance between [FP64/FP32/FP16/INT8,FP64/FP32/FP16/FP8/INT8] on NVIDIA [Ampere,Hopper] GPUs to boost the performance of output accuracy-preserving mixed-precision computation of Genome-Wide Association Studies (GWAS) of 305K patients from the UK BioBank, the largest-ever GWAS cohort studied for genetic epistasis using a multivariate approach. Tile-centric adaptive-precision linear algebraic techniques motivated by reducing data motion gain enhanced significance with low-precision GPU arithmetic. At the core of Kernel Ridge Regression (KRR) techniques for GWAS lie compute-bound cubic-complexity matrix operations that inhibit scaling to aspirational dimensions of the population, genotypes, and phenotypes. We accelerate KRR matrix generation by redesigning the computation for Euclidean distances to engage INT8 tensor cores while exploiting symmetry.We accelerate solution of the regularized KRR systems by deploying a new four-precision Cholesky-based solver, which, at 1.805 mixed-precision ExaOp/s on a nearly full Alps system, outperforms the state-of-the-art CPU-only REGENIE GWAS software by five orders of magnitude.

q-bio.GN↗

arXiv2024

Hi-SAM: A high-scalable authentication model for satellite-ground Zero-Trust system using mean field game

As more and more Internet of Thing (IoT) devices are connected to satellite networks, the Zero-Trust Architecture brings dynamic security to the satellite-ground system, while frequent authentication creates challenges for system availability. To make the system's accommodate more IoT devices, this paper proposes a high-scalable authentication model (Hi-SAM). Hi-SAM introduces the Proof-of-Work idea to authentication, which allows device to obtain the network resource based on frequency. To optimize the frequency, mean field game is used for competition among devices, which can reduce the decision space of large-scale population games. And a dynamic time-range message authentication code is designed for security. From the test at large population scales, Hi-SAM is superior in the optimization of authentication workload and the anomaly detection efficiency.

eess.SY↗

arXiv2024

Microscale physics and macroscale convective heat transfer in supercritical fluids

Driven by fundamental thermodynamic efficiency considerations, an emerging trend in the energy and propulsion systems is that the working fluid operates at a pressure above the critical pressure. Energy transport is thus accompanied by dramatic and strongly nonlinear variations of thermophysical properties, which cause abnormal heat transfer behavior and non-ideal gas effects. This situation raises a crucial challenge for the heat exchanger and turbomachinery design, overall energy and exergy efficiency. We aim to provide a multi-scale overview of the flow and thermal behavior of fluid above the critical point: microscopic physics and macroscopic transport. Microscopic physics, i.e. near-critical thermophysical properties, phase transition and fluid dynamics, are introduced. A particular focus will be on the supercritical pseudo boiling process, which is a generalization of classical liquid-vapor phase transitions to non-equilibrium supercritical states. These new perspectives lead to a revised view of the state space. Further, recent results demonstrated the possibility of stable supercritical fluid interfaces without surface tension. On the macroscale, recent progress on the modeling of turbulent flow and convective heat transfer of supercritical fluids are summarized. Direct numerical simulation is able to offer insights into the physics of thermal fluids. We start with a description of fundamental fluid mechanics problems related to supercritical fluids. In addition, the heat transfer deterioration in supercritical fluids is found to be closely connected to the flow relaminarization by the non-uniform body-force. Finally, various modeling approaches such as recently developed advanced Reynolds-averaged turbulence modeling as well as machine-learning methods are summarized.

physics.flu-dyn↗

arXiv2024

Identifying Semantic Induction Heads to Understand In-Context Learning

Although large language models (LLMs) have demonstrated remarkable performance, the lack of transparency in their inference logic raises concerns about their trustworthiness. To gain a better understanding of LLMs, we conduct a detailed analysis of the operations of attention heads and aim to better understand the in-context learning of LLMs. Specifically, we investigate whether attention heads encode two types of relationships between tokens present in natural languages: the syntactic dependency parsed from sentences and the relation within knowledge graphs. We find that certain attention heads exhibit a pattern where, when attending to head tokens, they recall tail tokens and increase the output logits of those tail tokens. More crucially, the formulation of such semantic induction heads has a close correlation with the emergence of the in-context learning ability of language models. The study of semantic attention heads advances our understanding of the intricate operations of attention heads in transformers, and further provides new insights into the in-context learning of LLMs.

cs.CL↗

arXiv2024

Copyright Protection in Generative AI: A Technical Perspective

Generative AI has witnessed rapid advancement in recent years, expanding their capabilities to create synthesized content such as text, images, audio, and code. The high fidelity and authenticity of contents generated by these Deep Generative Models (DGMs) have sparked significant copyright concerns. There have been various legal debates on how to effectively safeguard copyrights in DGMs. This work delves into this issue by providing a comprehensive overview of copyright protection from a technical perspective. We examine from two distinct viewpoints: the copyrights pertaining to the source data held by the data owners and those of the generative models maintained by the model builders. For data copyright, we delve into methods data owners can protect their content and DGMs can be utilized without infringing upon these rights. For model copyright, our discussion extends to strategies for preventing model theft and identifying outputs generated by specific models. Finally, we highlight the limitations of existing techniques and identify areas that remain unexplored. Furthermore, we discuss prospective directions for the future of copyright protection, underscoring its importance for the sustainable and ethical development of Generative AI.

cs.CR↗

arXiv2024

Harnessing Integrated CPU-GPU System Memory for HPC: a first look into Grace Hopper

Memory management across discrete CPU and GPU physical memory is traditionally achieved through explicit GPU allocations and data copy or unified virtual memory. The Grace Hopper Superchip, for the first time, supports an integrated CPU-GPU system page table, hardware-level addressing of system allocated memory, and cache-coherent NVLink-C2C interconnect, bringing an alternative solution for enabling a Unified Memory system. In this work, we provide the first in-depth study of the system memory management on the Grace Hopper Superchip, in both in-memory and memory oversubscription scenarios. We provide a suite of six representative applications, including the Qiskit quantum computing simulator, using system memory and managed memory. Using our memory utilization profiler and hardware counters, we quantify and characterize the impact of the integrated CPU-GPU system page table on GPU applications. Our study focuses on first-touch policy, page table entry initialization, page sizes, and page migration. We identify practical optimization strategies for different access patterns. Our results show that as a new solution for unified memory, the system-allocated memory can benefit most use cases with minimal porting efforts.

cs.DC↗

arXiv2024

Quasi-Nambu-Goldstone modes in many-body scar models

From the quasisymmetry-group perspective [Phys. Rev. Lett. 126, 120604 (2021)], we show the universal existence of collective, coherent modes of excitations in many-body scar models in the degenerate limit, where the energy spacing in the scar tower vanishes. The number of these modes, as well as the quantum numbers carried by them, are given, not by the symmetry of the Hamiltonian, but by the quasisymmetry of the scar tower: hence the name quasi-Nambu-Goldstone modes. Based on this, we draw a concrete analogy between the paradigm of spontaneous symmetry breaking and the many-body scar physics in the degenerate limit.

cond-mat.str-el↗

arXiv2024

Absence of measurement-induced entanglement transition due to feedback-induced skin effect

A quantum many-body system subject to unitary evolution and repeated local measurements with an increasing rate undergoes a measurement-induced entanglement transition from extensive (or subextensive) to area law entropy scaling. We find that certain open boundary systems under "generalized monitoring", consisting of "projective monitoring" and conditional feedback, display an anomalous late-time particle concentration on the edge, reminiscent of the "skin effect" in non-Hermitian systems. Such feedback-induced skin effect will suppress the entanglement generation, rendering the system short-range entangled without measurement-induced entanglement transition. While initially emerged in noninteracting models, such skin effect can also occur in chaotic interacting systems and Floquet quantum circuits subjected to random generalized measurements. Since the dynamics of the skin effect do not require post selection, and can be observed at the particle number level, the phenomenon is experimentally relevant and accessible in noisy intermediate-scale quantum platforms, such as trapped ions.

quant-ph↗