arXiv subjects

Ting Cao

Publications and source records attributed to Ting Cao.

At least 73 records · Page 4Linked to original sources

arXiv2025

AdaNav: Adaptive Reasoning with Uncertainty for Vision-Language Navigation

Vision Language Navigation (VLN) requires agents to follow natural language instructions by grounding them in sequential visual observations over long horizons. Explicit reasoning could enhance temporal consistency and perception action alignment, but reasoning at fixed steps often leads to suboptimal performance and unnecessary computation. To address this, we propose AdaNav, an uncertainty-based adaptive reasoning framework for VLN. At its core is the Uncertainty Adaptive Reasoning Block (UAR), a lightweight plugin that dynamically triggers reasoning. We introduce Action Entropy as a policy prior for UAR and progressively refine it through a Heuristics to RL training method, enabling agents to learn difficulty aware reasoning policies under the strict data limitations of embodied tasks. Results show that with only 6K training samples, AdaNav achieves substantial gains over closed source models trained on million scale data, improving success rate by 20% on R2R val-unseen, 11.7% on RxR-CE, and 11.4% in real world scenes. The code is available at https://github.com/xinding-sys/AdaNav.

cs.RO↗

arXiv2025

Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management

Large Language Models (LLMs) suffer from significant performance degradation when processing long contexts due to proactive interference, where irrelevant information in earlier parts of the context disrupts reasoning and memory recall. While most research focuses on external memory systems to augment LLMs' capabilities, we propose a complementary approach: empowering LLMs with Active Context Management (ACM) tools to actively sculpt their internal working memory. We introduce Sculptor, a framework that equips LLMs with three categories of tools: (1) context fragmentation, (2) summary, hide, and restore, and (3) precise search. Our approach enables LLMs to proactively manage their attention and working memory, analogous to how humans selectively focus on relevant information while filtering out distractions. Experimental evaluation on diverse long-context benchmarks demonstrates that Sculptor significantly improves performance even without specific training, leveraging LLMs' inherent tool-calling and instruction-following capabilities. To further optimize these strategies, we introduce a novel dynamic context-aware reinforcement learning (RL) approach, advancing the training of an agent that actively modifies its own conversational history. By enabling Active Context Management, Sculptor not only mitigates proactive interference but also provides a cognitive foundation for more reliable reasoning across diverse long-context tasks-highlighting that explicit context-control strategies, rather than merely larger token windows, are key to robustness at scale.

cs.CL↗

arXiv2025

Scaling LLM Test-Time Compute with Mobile NPU on Smartphones

Deploying Large Language Models (LLMs) on mobile devices faces the challenge of insufficient performance in smaller models and excessive resource consumption in larger ones. This paper highlights that mobile Neural Processing Units (NPUs) have underutilized computational resources, particularly their matrix multiplication units, during typical LLM inference. To leverage this wasted compute capacity, we propose applying parallel test-time scaling techniques on mobile NPUs to enhance the performance of smaller LLMs. However, this approach confronts inherent NPU challenges, including inadequate hardware support for fine-grained quantization and low efficiency in general-purpose computations. To overcome these, we introduce two key techniques: a hardware-aware tile quantization scheme that aligns group quantization with NPU memory access patterns, and efficient LUT-based replacements for complex operations such as Softmax and dequantization. We design and implement an end-to-end inference system that leverages the NPU's compute capability to support test-time scaling on Qualcomm Snapdragon platforms. Experiments show our approach brings significant speedups: up to 19.0 for mixed-precision GEMM and 2.2 for Softmax. More importantly, we demonstrate that smaller models using test-time scaling can match or exceed the accuracy of larger models, achieving a new performance-cost Pareto frontier.

cs.DC↗

arXiv2025

MVDRAM: Enabling GeMV Execution in Unmodified DRAM for Low-Bit LLM Acceleration

General matrix-vector multiplication (GeMV) remains a critical latency bottleneck in large language model (LLM) inference, even with quantized low-bit models. Processing-Using-DRAM (PUD), an analog in-DRAM computing technique, has the potential to repurpose on-device DRAM as a GeMV engine, offering additional high-throughput processing capabilities to widespread consumer devices without DRAM modifications. However, applying PUD to GeMV operations in the LLM inference pipeline incurs significant overheads $\textit{before}$ and $\textit{after}$ in-DRAM computation, diminishing the benefits of its high-throughput processing capabilities. This paper presents MVDRAM, the first practical system to accelerate GeMV operations for low-bit LLM inference using unmodified DRAM. By leveraging the data sharing patterns and mathematical linearity in GeMV operations, MVDRAM orchestrates the processor and DRAM to eliminate the costs associated with pre-arranging inputs and bit-transposition of outputs required in conventional PUD approaches. Our experimental evaluation with four DDR4 DRAM modules shows that MVDRAM achieves comparable or even better inference speed than the processor-based implementation for GeMV operations in low-bit (under 4-bit) LLM. In particular, MVDRAM achieves up to 7.29$\times$ speedup and 30.5$\times$ energy efficiency for low-bit GeMV operations. For end-to-end LLM inference, MVDRAM achieves 2.18$\times$ and 1.31$\times$ throughput improvements, along with 3.04$\times$ and 2.35$\times$ energy efficiency, for 2-bit and 4-bit quantized low-bit models, respectively. MVDRAM has the potential to redefine the AI hardware landscape by demonstrating the feasibility of standard DRAM as an LLM accelerator.

cs.AR↗

arXiv2025

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

Large language models (LLMs) are increasingly being deployed on mobile devices, but the limited DRAM capacity constrains the deployable model size. This paper introduces ActiveFlow, the first LLM inference framework that can achieve adaptive DRAM usage for modern LLMs (not ReLU-based), enabling the scaling up of deployable model sizes. The framework is based on the novel concept of active weight DRAM-flash swapping and incorporates three novel techniques: (1) Cross-layer active weights preloading. It uses the activations from the current layer to predict the active weights of several subsequent layers, enabling computation and data loading to overlap, as well as facilitating large I/O transfers. (2) Sparsity-aware self-distillation. It adjusts the active weights to align with the dense-model output distribution, compensating for approximations introduced by contextual sparsity. (3) Active weight DRAM-flash swapping pipeline. It orchestrates the DRAM space allocation among the hot weight cache, preloaded active weights, and computation-involved weights based on available memory. Results show ActiveFlow achieves the performance-cost Pareto frontier compared to existing efficiency optimization methods.

cs.LG↗

arXiv2025

TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge

Ternary quantization has emerged as a powerful technique for reducing both computational and memory footprint of large language models (LLM), enabling efficient real-time inference deployment without significantly compromising model accuracy. Conventional LLM inference platforms (e.g GPUs) cannot capitalize on its benefits, as they (i) lack native support for ternary arithmetic and memory specialization and (ii) remain severely under-utilized in low-batch, real-time scenarios. In this work, we propose TENET, a sparse-aware LUT-centric architecture that co-optimizes algorithm, compute, and memory for ternary LLM inference. To maximize the efficiency of Ternary Linear layer, TENET introduces a Sparse Ternary LUT (STL) core that optimizes ternary mixed-precision GEMM using a symmetric precompute lookup table. It also features Dynamic Activation N:M Sparsity to exploit the sparsity within the activation of each token. Additionally, we propose a LUT-based 64B:80B ternary weight decompression module to fully exploit the memory efficiency of ternary values. At the system level, we design a heterogeneous TENET accelerator with full programmability that integrates STL cores with high-precision cores. An associated Linear-Projection-aware Sparse Attention dataflow is introduced to optimize memory access and hardware utilization. We implement TENET accelerator prototype on both FPGA and ASIC platforms. Experiments across various model sizes and workloads demonstrate that TENET-FPGA and TENET-ASIC improve energy efficiency by 4.3$\times$ and 21.1$\times$, respectively, compared to the A100 GPU. Furthermore, TENET-ASIC achieves a 2.7$\times$ average speedup compared to the A100 GPU in end-to-end inference latency.

cs.AR↗

arXiv2025

Visualizing Electronic Structure of Twisted Bilayer MoTe2 in Devices

The pursuit of emergent quantum phenomena lies at the forefront of modern condensed matter physics. A recent breakthrough in this arena is the discovery of the fractional quantum anomalous Hall effect (FQAHE) in twisted bilayer MoTe2 (tbMoTe2), marking a paradigm shift and establishing a versatile platform for exploring the intricate interplay among topology, magnetism, and electron correlations. While significant progress has been made through both optical and electrical transport measurements, direct experimental insights into the electronic structure - crucial for understanding and modeling this system - have remained elusive. Here, using spatially and angle-resolved photoemission spectroscopy (μ-ARPES), we directly map the electronic band structure of tbMoTe2. We identify the valence band maximum, whose partial filling underlies the FQAHE, at the K points, situated approximately 150 meV above the Γ valley. By fine-tuning the doping level via in-situ alkali metal deposition, we also resolve the conduction band minimum at the K point, providing direct evidence that tbMoTe2 exhibits a direct band gap - distinct from all previously known moire bilayer transition metal dichalcogenide systems. These results offer critical insights for theoretical modeling and advance our understanding of fractionalized excitations and correlated topological phases in this emergent quantum material.

cond-mat.mes-hall↗

arXiv2025

Moiré excitons in generalized Wigner crystals

Moiré superlattices of transition-metal dichalcogenide bilayers host strong Coulomb interactions residing in narrow electron bands, leading to correlated insulating states at fractional carrier doping densities, known as generalized Wigner crystals. In excited states, the formation of moiré excitons is expected to be fundamentally shaped by the Wigner-crystal ground states, manifesting an intricate interplay between electronic and excitonic correlations. However, the microscopic description of these Wigner crystalline excitons (WCEs) remains elusive, largely subject to speculations, and is further needed for the understanding of exotic excitonic phases (e.g., exciton insulators and exciton density waves) and their unique properties (e.g., anomalous exciton diffusion). Here, using first-principles many-body GW-Bethe-Salpeter-equation calculations, we directly reveal the internal structures of WCEs in angle-aligned MoSe2/MoS2 moiré heterostructure at hole fillings of 1/3 and 2/3. Our results unveil the propagation of correlation effects from the ground state to excited states, shaping the real-space characteristics of WCEs. The strong two-particle excitonic correlations dominate over the kinetic energy of free electron-hole pairs, in analog to the strong single-particle correlations of flat bands. We propose that such unusual excited-state correlation effects of WCEs can be experimentally probed by photocurrent tunneling microscopy. Our work provides a microscopic understanding of strongly correlated WCEs, suggesting them as a highly tunable mixed boson-fermion platform to study many-body interactions and phenomena.

cond-mat.mtrl-sci↗

arXiv2025

StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition

With the rise of real-world human-AI interaction applications, such as AI assistants, the need for Streaming Video Dialogue is critical. To address this need, we introduce StreamMind, a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100) and enables proactive, always-on responses in real time, without explicit user intervention. To solve the key challenge of the contradiction between linear video streaming speed and quadratic transformer computation cost, we propose a novel perception-cognition interleaving paradigm named ''event-gated LLM invocation'', in contrast to the existing per-time-step LLM invocation. By introducing a Cognition Gate network between the video encoder and the LLM, LLM is only invoked when relevant events occur. To realize the event feature extraction with constant cost, we propose Event-Preserving Feature Extractor (EPFE) based on state-space method, generating a single perception token for spatiotemporal features. These techniques enable the video LLM with full-FPS perception and real-time cognition response. Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency, paving the way for ultra-high-FPS applications, such as Game AI and interactive media. The code and data is available at https://aka.ms/StreamMind.

cs.CV↗

arXiv2025

Optical Control of Integer and Fractional Chern Insulators

Optical control of topology, particularly in the presence of electron correlations, is a fascinating topic with broad scientific and technological impact. Twisted MoTe$_2$ bilayer (tMoTe$_2$) is a newly discovered zero-field fractional Chern insulator (FCI), exhibiting the fractionally quantized anomalous Hall (FQAH) effect. Since the chirality of the edge states and sign of the Chern number are determined by the underlying ferromagnetic polarization, manipulation of ferromagnetism would realize control of the CI/FCI states. Here, we demonstrate control and switching of ferromagnetic polarization, and thus the CI and FCI states by circularly polarized optical pumping in tMoTe$_2$. At low optical excitation power, we achieve on-demand preparation of ferromagnetic polarization by optical training, i.e., electrically tuning the system from non-ferromagnetic to desirable ferromagnetic states accompanied with helicity-selective optical pumping. With increased excitation power, we further realize direct optical switching of ferromagnetic polarization at a temperature far below the Curie temperature. Both optical training and direct switching of ferromagnetism are most effective near CI/FCI states, which we attribute to a gap enhanced valley polarization of photo-injected holes. We show that the magnetization can be dynamically switched by modulating the helicity of optical excitation. Spatially resolved measurements further demonstrate optical writing of a ferromagnetic, and thus a CI (or FCI) domain. Our work realizes precise optical control of a topological quantum many-body system with potential applications in topological spintronics, quantum memories, and creation of exotic edge states by programmable patterning of integer and fractional QAH domains.

cond-mat.mes-hall↗

arXiv2025

Twist-angle transferable continuum model and second flat Chern band in twisted MoTe2 and WSe2

We develop a twist-angle transferable continuum model for twisted transition metal dichalcogenide (tTMD) homobilayers, using tMoTe2 and tWSe2 as examples. All model parameters are extracted from density functional theory (DFT) calculations at a single twist angle (3.89°) and monolayer data. Our model captures both lattice relaxation effects and the long-range behavior of piezoelectric and ferroelectric potentials. Leveraging lattice relaxations obtained via machine learning force fields (MLFFs), the model can be efficiently transferred to other twist angles without requiring additional DFT calculations. It accurately reproduces the DFT band dispersions and quantum geometries across a wide range of twist angles. Furthermore, our model reveals that a second flat Chern band arises near 2° when the interlayer potential difference becomes comparable to the interlayer tunneling. This continuum model provides a clear understanding and starting point for engineering novel electronic phases in moiré TMDs through twist angles and lattice relaxations.

cond-mat.mtrl-sci↗

arXiv2025

Universal Magnetic Phases in Twisted Bilayer MoTe$_2$

Twisted bilayer MoTe$_2$ (tMoTe$_2$) has emerged as a robust platform for exploring correlated topological phases, notably supporting fractional Chern insulator (FCI) states at zero magnetic field across a wide range of twist angles. The evolution of magnetism and topology with twist angle remains an open question. Here, we systematically map the magnetic phase diagram of tMoTe$_2$ using local optical spectroscopy and scanning nanoSQUID-on-tip (nSOT) magnetometry. We identify spontaneous ferromagnetism at moiré filling factors $ν= -1$ and $-3$ over a twist angle range from 2.1$^\circ$ to 3.7$^\circ$, revealing a universal, twist-angle-insensitive ferromagnetic phase. At 2.1$^\circ$, we further observe robust ferromagnetism at $ν= -5$, absent in the devices with larger twist angle -- a signature of the flattening of higher bands in this twist angle range. Temperature-dependent measurements reveal a contrasting twist-angle dependence of the Curie temperatures between $ν= -1$ and $ν= -3$, indicating distinct interplay between exchange interaction and bandwidth for the two Chern bands. Despite spontaneous time-reversal symmetry breaking, we find no evidence of a topological gap at $ν= -3$; however, fragile correlated topological phases could be obscured by the device disorder evident in our spatially resolved measurements. Our results establish a global framework for understanding and controlling magnetic order in tMoTe$_2$ and highlight its potential for accessing correlated topological phases in higher energy Chern band.

cond-mat.mes-hall↗

arXiv2025

LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference

Large Language Model (LLM) inference becomes resource-intensive, prompting a shift toward low-bit model weights to reduce the memory footprint and improve efficiency. Such low-bit LLMs necessitate the mixed-precision matrix multiplication (mpGEMM), an important yet underexplored operation involving the multiplication of lower-precision weights with higher-precision activations. Off-the-shelf hardware does not support this operation natively, leading to indirect, thus inefficient, dequantization-based implementations. In this paper, we study the lookup table (LUT)-based approach for mpGEMM and find that a conventional LUT implementation fails to achieve the promised gains. To unlock the full potential of LUT-based mpGEMM, we propose LUT Tensor Core, a software-hardware co-design for low-bit LLM inference. LUT Tensor Core differentiates itself from conventional LUT designs through: 1) software-based optimizations to minimize table precompute overhead and weight reinterpretation to reduce table storage; 2) a LUT-based Tensor Core hardware design with an elongated tiling shape to maximize table reuse and a bit-serial design to support diverse precision combinations in mpGEMM; 3) a new instruction set and compilation optimizations for LUT-based mpGEMM. LUT Tensor Core significantly outperforms existing pure software LUT implementations and achieves a 1.44$\times$ improvement in compute density and energy efficiency compared to previous state-of-the-art LUT-based accelerators.

cs.AR↗

arXiv2025

SwarmThinkers: Learning Physically Consistent Atomic KMC Transitions at Scale

Can a scientific simulation system be physically consistent, interpretable by design, and scalable across regimes--all at once? Despite decades of progress, this trifecta remains elusive. Classical methods like Kinetic Monte Carlo ensure thermodynamic accuracy but scale poorly; learning-based methods offer efficiency but often sacrifice physical consistency and interpretability. We present SwarmThinkers, a reinforcement learning framework that recasts atomic-scale simulation as a physically grounded swarm intelligence system. Each diffusing particle is modeled as a local decision-making agent that selects transitions via a shared policy network trained under thermodynamic constraints. A reweighting mechanism fuses learned preferences with transition rates, preserving statistical fidelity while enabling interpretable, step-wise decision making. Training follows a centralized-training, decentralized-execution paradigm, allowing the policy to generalize across system sizes, concentrations, and temperatures without retraining. On a benchmark simulating radiation-induced Fe-Cu alloy precipitation, SwarmThinkers is the first system to achieve full-scale, physically consistent simulation on a single A100 GPU, previously attainable only via OpenKMC on a supercomputer. It delivers up to 4963x (3185x on average) faster computation with 485x lower memory usage. By treating particles as decision-makers, not passive samplers, SwarmThinkers marks a paradigm shift in scientific simulation--one that unifies physical consistency, interpretability, and scalability through agent-driven intelligence.

cs.AI↗

arXiv2025

SparStencil: Retargeting Sparse Tensor Cores to Scientific Stencil Computations via Structured Sparsity Transformation

Sparse Tensor Cores offer exceptional performance gains for AI workloads by exploiting structured 2:4 sparsity. However, their potential remains untapped for core scientific workloads such as stencil computations, which exhibit irregular sparsity patterns.This paper presents SparStencil, the first system to retarget sparse TCUs for scientific stencil computations through structured sparsity transformation. SparStencil introduces three key techniques: (1) Adaptive Layout Morphing, which restructures stencil patterns into staircase-aligned sparse matrices via a flatten-and-crush pipeline; (2) Structured Sparsity Conversion, which formulates transformation as a graph matching problem to ensure compatibility with 2:4 sparsity constraints; (3) Automatic Kernel Generation, which compiles transformed stencils into optimized sparse MMA kernels via layout search and table-driven memory mapping. Evaluated on 79 stencil kernels spanning diverse scientific domains, SparStencil achieves up to 7.1x speedup (3.1x on average) over state-of-the-art framework while reducing code complexity and matching or exceeding expert-tuned performance in both compute throughput and memory efficiency.

cs.CE↗

arXiv2025

SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention.

cs.LG↗

arXiv2025

Enhancing Shape Perception and Segmentation Consistency for Industrial Image Inspection

Semantic segmentation stands as a pivotal research focus in computer vision. In the context of industrial image inspection, conventional semantic segmentation models fail to maintain the segmentation consistency of fixed components across varying contextual environments due to a lack of perception of object contours. Given the real-time constraints and limited computing capability of industrial image detection machines, it is also necessary to create efficient models to reduce computational complexity. In this work, a Shape-Aware Efficient Network (SPENet) is proposed, which focuses on the shapes of objects to achieve excellent segmentation consistency by separately supervising the extraction of boundary and body information from images. In SPENet, a novel method is introduced for describing fuzzy boundaries to better adapt to real-world scenarios named Variable Boundary Domain (VBD). Additionally, a new metric, Consistency Mean Square Error(CMSE), is proposed to measure segmentation consistency for fixed components. Our approach attains the best segmentation accuracy and competitive speed on our dataset, showcasing significant advantages in CMSE among numerous state-of-the-art real-time segmentation networks, achieving a reduction of over 50% compared to the previously top-performing models.

cs.CV↗

arXiv2025

Geometric Origin of Phonon Magnetic Moment in Dirac Materials

We develop a theory for the phonon magnetic moment in doped Dirac materials, treating phonons as emergent gauge and gravitational fields coupled to Dirac fermions in curved space. By classifying electron-phonon coupling into angular momentum channels of Fermi surface deformation, we show that the phonon moment arises from two mechanisms: proportional to the electron Hall conductivity through the emergent gauge field coupling, and to the Hall viscosity through the frame field coupling. Applying our theory to Cd$_3$As$_2$ with first-principles calculations, we find quantitative agreement with experiment. Our results reveal a general mechanism for dynamically generating large phonon magnetism in metals and suggest a new route for probing Hall viscosity via phonon dynamics.

cond-mat.mtrl-sci↗