arXiv subjects

Qijia He

Publications and source records attributed to Qijia He.

12 recordsLinked to original sources

arXiv2026

DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.

cs.AI↗

arXiv2026

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by $8.6\times$ and, from a base initialization, reaches $21.3\%$ held-out math accuracy in $2.6\times$ fewer optimizer updates than OPD; both methods finish near $22\%$. It improves best-observed performance by $4.8$ and $4.1$ percentage points on ALFWorld with LoRA and agentic code, respectively. Ablations show that segment-averaged triggers outperform token-spike triggers. Behavior cloning is also more effective than the tested OPD surrogate for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.

cs.LG↗

arXiv2026

PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that provides annealed process supervision over reasoning and tool-call components. To reduce over-reliance on the training-only trace context, PACT incorporates prompt-only anchoring with standard prompt-only RL updates. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently outperforms strong SFT- and RL-based baselines. Further analyses support the complementarity of the two training objectives and the benefits of component-aware supervision and prompt-only anchoring. These results highlight the effectiveness of privileged trace co-training for multi-turn tool-use learning. The project page is available at https://zhenbangdu.github.io/pact-project-page.

cs.CL↗

arXiv2026

Multi-Harness Reinforcement Learning for Coding Agents: Training Gains and Harness Dependence

Recent systems train coding agents with reinforcement learning (RL) on trajectories collected through multiple execution harnesses. However, gains under the training harnesses leave open whether the improvements transfer to unseen interfaces and how they depend on reward normalization. We present a controlled study that measures RL gains under fixed harnesses and compares the same models across harnesses. We train two Qwen3-8B RL variants from the same supervised fine-tuning (SFT) checkpoint using identical replay data. They differ only in whether rewards for each task are normalized within each harness (Within) or across harnesses (Cross). On SWE-bench Verified, we identify three main findings: (i) Multi-harness RL yields modest gains over the SFT baseline under both normalization rules. Across the four training harnesses, the estimated mean improvements are +0.48 and +0.77 percentage points for Within and Cross, respectively. (ii) Cross increases the fraction of trajectories with nonzero advantage but provides no consistent advantage over Within on an unseen harness across training seeds and evaluation budgets; neither rule significantly improves over SFT on this harness. (iii) Performance varies substantially more across harnesses than with RL training. After averaging over model variants, the highest and lowest source-harness solve rates differ by 7.13 percentage points, far exceeding the estimated training gains. Even after multi-harness RL, the observed performance differences across execution interfaces remain substantially larger than either the gains from training or the differences between normalization rules.

cs.AI↗

arXiv2026

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from $O(ε^{-4})$ to $O(ε^{-2})$ as $ε\to0$, where $1+ε$ is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as $G^{-2/5}$ after tuning the step size, where $G$ is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as $G\to\infty$, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays.

cs.LG↗

arXiv2026

Provable Test-Time Scaling for Beam Search in LLM Reasoning

Beam-search-based test-time methods provide an effective way to improve large language model (LLM) performance on long-horizon generation by pruning invalid reasoning paths early, leading to significantly improved reasoning efficiency and more favorable test-time cost scaling. Despite strong empirical success, the theoretical understanding of beam search remains limited. In this paper, we study the test-time compute guarantee of the commonly used beam search framework that uses the model's internal log-likelihood for intermediate scoring, while relying on an external reward model only after a complete response is generated. We first establish a lower bound for vanilla beam search, showing that at least $Ω(C^\star(x)^2)$ samples are required for the optimal response to survive, where $C^\star(x)$ is the token-level coverage coefficient for prompt $x$. This motivates our modified confidence-filtered beam search (CF-Beam), which reduces the sufficient coverage dependence from quadratic to nearly linear under prefix competitiveness, for fixed horizon, gap, and target accuracy. We then show that the regret of CF-Beam is upper-bounded by the probability of rare failure events and the reward estimation error scaled by a path-level coverage coefficient, where the rare-failure term vanishes as per-step sampling increases. Our results highlight a fundamental advantage of beam search over sequence-level inference methods such as Best-of-N and Best-of-Majority. While the guarantees of these approaches typically involve coverage coefficients that grow exponentially with the horizon $L$, CF-Beam controls the dominant search-induced term through a token-level coverage coefficient that scales polynomially with $L$. Our numerical experiments further confirm that beam search is more robust on hard instances and under increasing reasoning horizons.

cs.LG↗

arXiv2026

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Scalable Vector Graphics (SVG) are essential for technical illustration and digital design, offering resolution independence and semantic editability. In practice, original vector files are frequently lost, leaving only rasterized versions (e.g., PNG, JPEG) that resist modification, while manual reconstruction is prohibitively expensive. Progress on automating raster-to-SVG conversion has been bottlenecked by two gaps: existing SVG datasets are dominated by icons and decorative graphics that lack the complexity of professional diagrams, and existing benchmarks rely on pixel- or embedding-level similarity that fails to capture structural correctness (e.g., broken connectivity, misplaced arrows). We close both gaps with paired contributions targeting diagram-centric figures (e.g., model architectures, flowcharts, schematics). For training, we introduce VFIG-Data, the largest figure-to-SVG dataset of its kind at 66K pairs, combining real paper figures converted via a describe-and-generate pipeline with programmatic diagrams that supply noise-free supervision over arrow styles, fonts, and geometry. For evaluation, we introduce VFIG-Bench, a structure-aware evaluation suite, paired with VFIG-Bench-OOD, an out-of-distribution set of figures manually curated from highly cited arXiv papers. Beyond pixel and embedding similarity, our protocol reports rubric-based VLM-Judge scores and Elo ratings from pairwise human preference evaluation. Built on these contributions, VFIG is a VLM family trained with a simple-to-complex SFT curriculum followed by RL with rendering-aware rewards. VFIG achieves state-of-the-art open-source performance, outperforming the best open-source VLM baseline by over 30%, and matches Claude Sonnet 4.6 on VFIG-BENCH: Gemini-Judge 78.2% vs. 76.7% and GPT-Judge 87.5% vs. 87.4%. It remains slightly behind the strongest proprietary models GPT-5.2 and Gemini-3.

cs.CV↗

arXiv2026

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer. Existing cost-aware systems typically treat such failures as cascade decisions: try a cheap model first, then escalate hard cases to a stronger and more expensive model. In coding, however, execution feedback can also make further cheap-model recovery worthwhile, raising a budgeted deployment question: when should an agent spend more cheap compute, and when should it escalate? We formulate this post-failure decision as recovery routing over heterogeneous actions and train a supervised router from execution rollouts. To make the same router usable under changing budgets, we add a Conformal Risk Control (CRC) layer that selects a deployment-time cost penalty without retraining and provides marginal expected-cost control under exchangeability. Across held-out failures from five coding benchmarks, cheap recovery and escalation exhibit complementary success patterns. The calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade baseline; in the main GPT-5.4-nano/GPT-5.4 setting, one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost. Code is available at https://github.com/Qijia-He/agent-budget-control.

cs.AI↗

arXiv2026

Dealing with positivity violations in mediation analysis via weighted controlled effects, with application to assessing immune correlates of protection in antigen-experienced participants

Causal mediation analysis has become an important and increasingly used framework for evaluating candidate immune response biomarkers in vaccine research. A controlled effects approach has been proposed to estimate controlled risk curves under a counterfactual scenario in which the entire study population is vaccinated and their post-vaccination immune responses are set to a range of fixed levels. This framework performs well when the study population is antigenically naïve, that is, individuals have not been previously exposed to the antigen, as is common in HIV-1 vaccine research and during the early phases of the COVID-19 pandemic. However, the controlled effects framework becomes more challenging to apply in antigen-experienced populations, where prior vaccination or infection has occurred, as in the case of influenza, dengue, and more recent phases of the COVID-19 pandemic. In such settings, a key identification assumption for valid causal mediation analysis, the positivity assumption, is violated: it is no longer plausible to conceive of a hypothetical intervention that sets a post-vaccination immune marker to a fixed level below an individual's baseline immune level. In this article, we introduce a weighted controlled risk approach that targets a subpopulation for whom there is a prespecified probability of attaining a post-vaccination immune marker level. We further generalize this framework to study contrasts of controlled risks for relevant subpopulations. We demonstrate the validity of the proposed estimators through simulation studies and apply the method to reanalyze post-vaccination neutralizing antibody titers against Omicron BA.4/BA.5 as an immune correlate of COVID-19 in the Coronavirus Variant Immunologic Landscape (COVAIL) trial. R code to implement the proposed method can be found on Github: https://github.com/Qijia-He/weighted_CVE.

stat.ME↗

arXiv2025

Rethinking Human Preference Evaluation of LLM Rationales

Large language models (LLMs) often generate natural language rationales -- free-form explanations that help improve performance on complex reasoning tasks and enhance interpretability for human users. However, evaluating these rationales remains challenging. While recent work has relied on binary preference judgments from humans or LLM judges, such evaluations are often opaque and coarse-grained, offering limited insight into what makes one rationale better than another. In this work, we rethink preference evaluation for LLM-generated rationales by asking: (1) What attributes define good rationales? (2) Can human preferences be explained by these attributes? (3) Can attribute-based evaluation overcome the limitations of binary comparisons? We identify a set of key rationale attributes from prior literature and assess them using automatic metrics, LLM judgments, and human annotations. We then analyze two standard human preference datasets MT Bench and Chatbot Arena using SHAP to identify which attributes best explain human preference outcomes. Finally, we re-evaluate model-generated rationales using attribute-specific ELO scores, revealing more nuanced model comparisons and insights. Our findings suggest that fine-grained attribute evaluations can better characterize rationale quality and guide future research toward more interpretable and reliable evaluation practices.

cs.AI↗

arXiv2023

Generalizing the intention-to-treat effect of an active control against placebo from historical placebo-controlled trials to an active-controlled trial: A case study of the efficacy of daily oral TDF/FTC in the HPTN 084 study

In many clinical settings, an active-controlled trial design (e.g., a non-inferiority or superiority design) is often used to compare an experimental medicine to an active control (e.g., an FDA-approved, standard therapy). One prominent example is a recent phase 3 efficacy trial, HIV Prevention Trials Network Study 084 (HPTN 084), comparing long-acting cabotegravir, a new HIV pre-exposure prophylaxis (PrEP) agent, to the FDA-approved daily oral tenofovir disoproxil fumarate plus emtricitabine (TDF/FTC) in a population of heterosexual women in 7 African countries. One key complication of interpreting study results in an active-controlled trial like HPTN 084 is that the placebo arm is not present and the efficacy of the active control (and hence the experimental drug) compared to the placebo can only be inferred by leveraging other data sources. \bz{In this article, we study statistical inference for the intention-to-treat (ITT) effect of the active control using relevant historical placebo-controlled trials data under the potential outcomes (PO) framework}. We highlight the role of adherence and unmeasured confounding, discuss in detail identification assumptions and two modes of inference (point versus partial identification), propose estimators under identification assumptions permitting point identification, and lay out sensitivity analyses needed to relax identification assumptions. We applied our framework to estimating the intention-to-treat effect of daily oral TDF/FTC versus placebo in HPTN 084 using data from an earlier Phase 3, placebo-controlled trial of daily oral TDF/FTC (Partners PrEP).

stat.ME↗

arXiv2022

NCAGC: A Neighborhood Contrast Framework for Attributed Graph Clustering

Attributed graph clustering is one of the most fundamental tasks among graph learning field, the goal of which is to group nodes with similar representations into the same cluster without human annotations. Recent studies based on graph contrastive learning method have achieved remarkable results when exploit graph-structured data. However, most existing methods 1) do not directly address the clustering task, since the representation learning and clustering process are separated; 2) depend too much on data augmentation, which greatly limits the capability of contrastive learning; 3) ignore the contrastive message for clustering tasks, which adversely degenerate the clustering results. In this paper, we propose a Neighborhood Contrast Framework for Attributed Graph Clustering, namely NCAGC, seeking for conquering the aforementioned limitations. Specifically, by leveraging the Neighborhood Contrast Module, the representation of neighbor nodes will be 'push closer' and become clustering-oriented with the neighborhood contrast loss. Moreover, a Contrastive Self-Expression Module is built by minimizing the node representation before and after the self-expression layer to constraint the learning of self-expression matrix. All the modules of NCAGC are optimized in a unified framework, so the learned node representation contains clustering-oriented messages. Extensive experimental results on four attributed graph datasets demonstrate the promising performance of NCAGC compared with 16 state-of-the-art clustering methods. The code is available at https://github.com/wangtong627/NCAGC.

cs.CV↗