arXiv subjects

Kai Yang

Publications and source records attributed to Kai Yang.

At least 91 records · Page 5Linked to original sources

arXiv2026

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST

cs.CL↗

arXiv2026

Machine Learning Model Trading with Verification under Information Asymmetry

Machine learning (ML) model trading, known for its role in protecting data privacy, faces a major challenge: information asymmetry. This issue can lead to model deception, a problem that current literature has not fully solved, where the seller misrepresents model performance to earn more. We propose a game-theoretic approach, adding a verification step in the ML model market that lets buyers check model quality before buying. However, this method can be expensive and offers imperfect information, making it harder for buyers to decide. Our analysis reveals that a seller might probabilistically conduct model deception considering the chance of model verification. This deception probability decreases with the verification accuracy and increases with the verification cost. To maximize seller payoff, we further design optimal pricing schemes accounting for heterogeneous buyers' strategic behaviors. Interestingly, we find that reducing information asymmetry benefits both the seller and buyer. Meanwhile, protecting buyer order information doesn't improve the payoff for the buyer or the seller. These findings highlight the importance of reducing information asymmetry in ML model trading and open new directions for future research.

cs.GT↗

arXiv2026

KDCM: Reducing Hallucination in LLMs through Explicit Reasoning Structures

To mitigate hallucinations in large language models (LLMs), we propose a framework that focuses on errors induced by prompts. Our method extends a chain-style knowledge distillation approach by incorporating a programmable module that guides knowledge graph exploration. This module is embedded as executable code within the reasoning prompt, allowing the model to leverage external structured knowledge during inference. Based on this design, we develop an enhanced distillation-based reasoning framework that explicitly regulates intermediate reasoning steps, resulting in more reliable predictions. We evaluate the proposed approach on multiple public benchmarks using GPT-4 and LLaMA-3.3. Experimental results show that code-guided reasoning significantly improves contextual modeling and reduces prompt-induced hallucinations. Specifically, HIT@1, HIT@3, and HIT@5 increase by 15.64%, 13.38%, and 13.28%, respectively, with scores exceeding 95% across several evaluation settings. These findings indicate that the proposed method effectively constrains erroneous reasoning while improving both accuracy and interpretability.

cs.CL↗

arXiv2026

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

Direct Preference Learning has emerged as a dominant offline paradigm for preference optimization. Most of these methods are based on the Bradley-Terry (BT) model for pairwise preference ranking, which directly aligns language model with human preference. Prior work has observed a counter-intuitive phenomenon termed likelihood displacement, where the absolute probability of preferred responses decreases simultaneously during training. We demonstrate that such displacement can lead to a more devastating failure mode, which we defined as \textit{Catastrophic Preference Shift}, where the lost preference probability mass inadvertently shifts toward out-of-distribution (OOD) responses. Such a failure mode is a key limitation shared across BT-style direct preference learning methods, due to the fundamental conflict between the unconstrained discriminative alignment and generative foundational capabilities, ultimately leading to severe performance degradation (e.g., SimPO suffers a significant drop in reasoning accuracy from 73.5\% to 37.5\%). We analyze existing BT-style methods from the probability evolution perspective and theoretically prove that these methods exhibit over-reliance on model initialization and can lead to preference shift. To resolve these counter-intuitive behaviors, we propose a theoretically grounded Stable Preference Optimization (SPO) framework that constrains preference learning within a safe alignment region. Empirical evaluations demonstrate that SPO effectively stabilizes and enhances the performance of existing BT-style preference learning methods. SPO provides new insights into the design of preference learning objectives and opens up new avenues towards more reliable and interpretable language model alignment.

cs.AI↗

arXiv2026

Mitigating Prompt-Induced Hallucinations in Large Language Models via Structured Reasoning

To address hallucination issues in large language models (LLMs), this paper proposes a method for mitigating prompt-induced hallucinations. Building on a knowledge distillation chain-style model, we introduce a code module to guide knowledge-graph exploration and incorporate code as part of the chain-of-thought prompt, forming an external knowledge input that provides more accurate and structured information to the model. Based on this design, we develop an improved knowledge distillation chain-style model and leverage it to analyze and constrain the reasoning process of LLMs, thereby improving inference accuracy. We empirically evaluate the proposed approach using GPT-4 and LLaMA-3.3 on multiple public datasets. Experimental results demonstrate that incorporating code modules significantly enhances the model's ability to capture contextual information and effectively mitigates prompt-induced hallucinations. Specifically, HIT@1, HIT@3, and HIT@5 improve by 15.64%, 13.38%, and 13.28%, respectively. Moreover, the proposed method achieves HIT@1, HIT@3, and HIT@5 scores exceeding 95% across several evaluation settings. These results indicate that the proposed approach substantially reduces hallucination behavior while improving the accuracy and verifiability of large language models.

cs.CL↗

arXiv2026

Unveiling Fiber Networks and Core Formation in the DR21 South Filament

We present high-resolution ($\sim$1000 AU) 3 mm observations with the NOrthern Extended Millimeter Array toward the DR21 South Filament, aiming to reveal its internal fragmentation and search for deeply embedded star-forming activities. Both the continuum and molecular line emissions align well with the filament axis traced by the low-resolution ($\sim$18$^{\prime\prime}$) column density map. The 3 mm continuum, CS (2$-$1), and HCO$^+$ (1$-$0) emissions reveal continuous and diffuse structures with measured FWHM widths of 0.054, 0.029, and 0.030 pc, respectively. In contrast, the H$^{13}$CO$^+$ (1$-$0) emission appears more clumpy and localized. The non-thermal motion in the filament is predominantly subsonic to transonic. We detect 13 dense cores in NH$_2$D (1$_{11}-1_{01}$), three of which coincide with continuum peaks; virial analysis suggests most are gravitationally bound. Using a friend-of-friend algorithm, we identify 32, 34, and 22 velocity-coherent fibers from the CS, HCO$^+$, and H$^{13}$CO$^+$ data, respectively. Compared to fibers traced by CS and HCO$^+$, H$^{13}$CO$^+$ fibers are more frequently associated with NH$_2$D cold cores and exhibit higher average mass-per-unit-length values. Differences among CS, HCO$^+$, and H$^{13}$CO$^+$ emissions likely arise from variations in effective critical densities. These results are consistent with a hierarchical structure, in which the 3.6-pc DR21SF contains velocity-coherent fibers and gravitationally bound dense cores.

astro-ph.GA↗

arXiv2025

SMC-X: A Distributed Scalable Monte Carlo Simulation Method for Chemically Complex Alloys

To predict the complex chemical evolution in multicomponent alloys, it is highly desirable to have accurate atomistic simulation methods capable of reaching sufficiently large spatial and temporal scales. In this work, we advance the recently proposed SMC-X method through distributed computation on either GPUs or CPUs, pushing both spatial and temporal scales of atomistic simulation of chemically complex alloys to previously inaccessible scales. This includes a record-breaking 128-billion-atom HEA system extending to the micrometer regime in space, and a 1-billion-atom HEA evolved over more than three million Monte Carlo swap steps, approaching the minute regime in time. We show that such large-scale simulations are essential for bridging the gap between experimental observations and theoretical predictions of the nanoprecipitate sizes in HEAs, based on analysis using the Lifshitz-Slyozov-Wagner (LSW) theory for diffusion-controlled coarsening. This work demonstrates the great potential of SMC-X for simulation-driven exploration of the chemical complexity in high-entropy materials at large spatial and temporal scales.

cond-mat.mtrl-sci↗

arXiv2025

Transferable potential for molecular dynamics simulations of borosilicate glasses and structural comparison of machine learning optimized parameters

The simulation of borosilicate glasses is challenging due to the composition and temperature dependent coordination state of boron atoms. Here, we present a newly developed machine learning optimized classical potential for molecular dynamics simulations that achieves transferability across diverse borosilicate glass compositions. Our potential accurately predicts the glass structural variations in short- and medium-range order in different glass compositions, including validating our potential against experimental X-ray structure factor data. Notably, these data are not included in the optimization framework, which focuses exclusively on density and four-fold coordinated boron fraction. We further investigate the impact of empirical parameters in the force field formulation on the microscopic bond lengths, bond angles and the macroscopic densities, providing new insights into the relationship between interatomic potentials and bulk glass behaviors.

cond-mat.dis-nn↗

arXiv2025

RoS-Guard: Robust and Scalable Online Change Detection with Delay-Optimal Guarantees

Online change detection (OCD) aims to rapidly identify change points in streaming data and is critical in applications such as power system monitoring, wireless network sensing, and financial anomaly detection. Existing OCD methods typically assume precise system knowledge, which is unrealistic due to estimation errors and environmental variations. Moreover, existing OCD methods often struggle with efficiency in large-scale systems. To overcome these challenges, we propose RoS-Guard, a robust and optimal OCD algorithm tailored for linear systems with uncertainty. Through a tight relaxation and reformulation of the OCD optimization problem, RoS-Guard employs neural unrolling to enable efficient parallel computation via GPU acceleration. The algorithm provides theoretical guarantees on performance, including expected false alarm rate and worst-case average detection delay. Extensive experiments validate the effectiveness of RoS-Guard and demonstrate significant computational speedup in large-scale system scenarios.

cs.LG↗

arXiv2025

Blow-up in finite or infinite time of the 2D cubic Zakharov-Kuznetsov equation

We prove that near-threshold negative energy solutions to the 2D cubic ($L^2$-critical) focusing Zakharov-Kuznetsov (ZK) equation blow-up in finite or infinite time. The proof consists of several steps. First, we show that if the blow-up conclusion is false, there are negative energy solutions arbitrarily close to the threshold that are globally bounded in $H^1$ and are spatially localized, uniformly in time. In the second step, we show that such solutions must in fact be exact remodulations of the ground state, and hence, have zero energy, which is a contradiction. This second step, a nonlinear Liouville theorem, is proved by contradiction, with a limiting argument producing a nontrivial solution to a (linear) linearized ZK equation obeying uniform-in-time spatial localization. Such nontrivial linear solutions are excluded by a local-viral space-time estimate. The general framework of the argument is modeled on Merle [29] and Martel & Merle [24], who treated the 1D problem of the $L^2$-critical gKdV equation. Several new features are introduced here to handle the 2D ZK case.

math.AP↗

arXiv2025

CE-Nav: Flow-Guided Reinforcement Refinement for Cross-Embodiment Local Navigation

Generalizing local navigation policies across diverse robot morphologies is a critical challenge. Progress is often hindered by the need for costly and embodiment-specific data, the tight coupling of planning and control, and the "disastrous averaging" problem where deterministic models fail to capture multi-modal decisions (e.g., turning left or right). We introduce CE-Nav, a novel two-stage (IL-then-RL) framework that systematically decouples universal geometric reasoning from embodiment-specific dynamic adaptation. First, we train an embodiment-agnostic General Expert offline using imitation learning. This expert, a conditional normalizing flow model named VelFlow, learns the full distribution of kinematically-sound actions from a large-scale dataset generated by a classical planner, completely avoiding real robot data and resolving the multi-modality issue. Second, for a new robot, we freeze the expert and use it as a guiding prior to train a lightweight, Dynamics-Aware Refiner via online reinforcement learning. This refiner rapidly learns to compensate for the target robot's specific dynamics and controller imperfections with minimal environmental interaction. Extensive experiments on quadrupeds, bipeds, and quadrotors show that CE-Nav achieves state-of-the-art performance while drastically reducing adaptation cost. Successful real-world deployments further validate our approach as an efficient and scalable solution for building generalizable navigation systems. Code is available at https://github.com/amap-cvlab/CE-Nav.

cs.RO↗

arXiv2025

From Sequence to Structure: Uncovering Substructure Reasoning in Transformers

Recent studies suggest that large language models (LLMs) possess the capability to solve graph reasoning tasks. Notably, even when graph structures are embedded within textual descriptions, LLMs can still effectively answer related questions. This raises a fundamental question: How can a decoder-only Transformer architecture understand underlying graph structures? To address this, we start with the substructure extraction task, interpreting the inner mechanisms inside the transformers and analyzing the impact of the input queries. Specifically, through both empirical results and theoretical analysis, we present Induced Substructure Filtration (ISF), a perspective that captures the substructure identification in the multi-layer transformers. We further validate the ISF process in LLMs, revealing consistent internal dynamics across layers. Building on these insights, we explore the broader capabilities of Transformers in handling diverse graph types. Specifically, we introduce the concept of thinking in substructures to efficiently extract complex composite patterns, and demonstrate that decoder-only Transformers can successfully extract substructures from attributed graphs, such as molecular graphs. Together, our findings offer a new insight on how sequence-based Transformers perform the substructure extraction task over graph data.

cs.CL↗

arXiv2025

Optimization of Transferable Interatomic Potentials for Glasses toward Experimental Properties

The accuracy of molecular simulations is fundamentally limited by the interatomic potentials that govern atomic interactions. Traditional potential development, which relies heavily on ab initio calculations, frequently struggles to reproduce the experimentally observed properties that govern real material behavior. To address this challenge, we present a machine learning-driven, active-learning optimization framework for optimizing classical interatomic potentials to reproduce experimental properties. Our method, here showcased on soda-lime borosilicate glasses, targets both global (density) and local (boron coordination) structural features across a wide range of compositions. By combining a surrogate model with iterative active learning, the framework efficiently explores a five-dimensional parameter space using only 400 molecular dynamics simulations over 17 iterations, making it highly data-efficient and eliminating the need for extensive simulation campaigns. Two transferable parameter sets are identified, each demonstrating good agreement with experimental measurements, including glass density, fraction of four-fold boron, and X-ray structure factor. The framework effectively captures and manages inherent trade-offs between structural objectives and compositional regimes, providing insights into the coordination behavior of boron in complex glass networks. The resulting classical force fields are generalizable and do not require reparameterization for individual compositions. Altogether, this work offers a scalable and experimentally grounded approach for developing transferable interatomic potentials suitable for a broad range of materials, including multi-component glass systems, and beyond.

cond-mat.dis-nn↗

arXiv2025

Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning

Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which uses a simple but effective method, an adaptive loss function, to mitigate training fluctuation and token inefficiency. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.

cs.CL↗

arXiv2025

A Self-Adaptive Frequency Domain Network for Continuous Intraoperative Hypotension Prediction

Intraoperative hypotension (IOH) is strongly associated with postoperative complications, including postoperative delirium and increased mortality, making its early prediction crucial in perioperative care. While several artificial intelligence-based models have been developed to provide IOH warnings, existing methods face limitations in incorporating both time and frequency domain information, capturing short- and long-term dependencies, and handling noise sensitivity in biosignal data. To address these challenges, we propose a novel Self-Adaptive Frequency Domain Network (SAFDNet). Specifically, SAFDNet integrates an adaptive spectral block, which leverages Fourier analysis to extract frequency-domain features and employs self-adaptive thresholding to mitigate noise. Additionally, an interactive attention block is introduced to capture both long-term and short-term dependencies in the data. Extensive internal and external validations on two large-scale real-world datasets demonstrate that SAFDNet achieves up to 97.3\% AUROC in IOH early warning, outperforming state-of-the-art models. Furthermore, SAFDNet exhibits robust predictive performance and low sensitivity to noise, making it well-suited for practical clinical applications.

cs.LG↗

arXiv2025

Transfer Learning and Machine Learning for Training Five Year Survival Prognostic Models in Early Breast Cancer

Prognostic information is essential for decision-making in breast cancer management. Recently trials have predominantly focused on genomic prognostication tools, even though clinicopathological prognostication is less costly and more widely accessible. Machine learning (ML), transfer learning and ensemble integration offer opportunities to build robust prognostication frameworks. We evaluate this potential to improve survival prognostication in breast cancer by comparing de-novo ML, transfer learning from a pre-trained prognostic tool and ensemble integration. Data from the MA.27 trial was used for model training, with external validation on the TEAM trial and a SEER cohort. Transfer learning was applied by fine-tuning the pre-trained prognostic tool PREDICT v3, de-novo ML included Random Survival Forests and Extreme Gradient Boosting, and ensemble integration was realized through a weighted sum of model predictions. Transfer learning, de-novo RSF, and ensemble integration improved calibration in MA.27 over the pre-trained model (ICI reduced from 0.042 in PREDICT v3 to <=0.007) while discrimination remained comparable (AUC increased from 0.738 in PREDICT v3 to 0.744-0.799). Invalid PREDICT v3 predictions were observed in 23.8-25.8% of MA.27 individuals due to missing information. In contrast, ML models and ensemble integration could predict survival regardless of missing information. Across all models, patient age, nodal status, pathological grading and tumor size had the highest SHAP values, indicating their importance for survival prognostication. External validation in SEER, but not in TEAM, confirmed the benefits of transfer learning, RSF and ensemble integration. This study demonstrates that transfer learning, de-novo RSF, and ensemble integration can improve prognostication in situations where relevant information for PREDICT v3 is lacking or where a dataset shift is likely.

cs.LG↗

arXiv2025

Matrix-variate integer-valued autoregressive processes

In the fields of sociology and economics, the modeling of matrix-variate integervalued time series is urgent. However, no prior studies have addressed the modeling of such data. To address this topic, this paper proposes a novel matrix-variate integer-valued autoregressive model. The key techniques lie in defining two leftand right-matricial thinning operators. The probabilistic and statistical properties of the proposed model are investigated. Furthermore, two estimation methods are developed: projection estimation and iterative least squares estimation. The corresponding asymptotic properties of these estimators are established. Additionally, the order-determination problem is addressed. In the simulation studies, the estimation results are given and the theoretical properties are verified. Finally, it is shown that the matrix-variate integer-valued autoregressive model is superior to the continuous matrix-variate autoregressive and multivariate integer-valued autoregressive models for matrix-variate integer-valued time series data.

math.ST↗

arXiv2025

Contributions to Robust and Efficient Methods for Analysis of High Dimensional Data

A ubiquitous feature of data of our era is their extra-large sizes and dimensions. Analyzing such high-dimensional data poses significant challenges, since the feature dimension is often much larger than the sample size. This thesis introduces robust and computationally efficient methods to address several common challenges associated with high-dimensional data. In my first manuscript, I propose a coherent approach to variable screening that accommodates nonlinear associations. I develop a novel variable screening method that transcends traditional linear assumptions by leveraging mutual information, with an intended application in neuroimaging data. This approach allows for accurate identification of important variables by capturing nonlinear as well as linear relationships between the outcome and covariates. Building on this foundation, I develop new optimization methods for sparse estimation using nonconvex penalties in my second manuscript. These methods address notable challenges in current statistical computing practices, facilitating computationally efficient and robust analyses of complex datasets. The proposed method can be applied to a general class of optimization problems. In my third manuscript, I contribute to robust modeling of high-dimensional correlated observations by developing a mixed-effects model based on Tsallis power-law entropy maximization and discussed the theoretical properties of such distribution. This model surpasses the constraints of conventional Gaussian models by accommodating a broader class of distributions with enhanced robustness to outliers. Additionally, I develop a proximal nonlinear conjugate gradient algorithm that accelerates convergence while maintaining numerical stability, along with rigorous statistical properties for the proposed framework.

math.ST↗