Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.08875

An Empirical Study of Agent Skills' Downstream Utility

Yu Cheng·Dehai Zhao·Zhongxin Liu·Qing Huang·Zhenchang Xing·Xiaoxue Ren
View original source ↗Read source document ↗Export metadata ↓

Abstract

Agent Skills package procedural guidance and resources for reuse, but a relevant Skill does not necessarily improve task performance. Existing studies characterize Skill content and evaluate downstream performance, yet provide limited explanations of how utility depends on content, execution configuration, and multi-Skill organization. We conduct an empirical study on 87 SkillsBench tasks, defining downstream utility as the pass-rate difference from No-Skill on the same tasks under the same model--harness configuration. We compare the same Skills across nine configurations, then examine alternative published Skills and organizations of fixed Skill sets under three selected configurations. We retrieve marketplace candidates from a curated corpus of 37,596 Skills. LLM-assisted analysis of content, execution traces, and final artifacts, followed by author review, relates provided support to actual use and task outcomes. The same Skills help some configurations and hurt others on 36.78\% of tasks, with trajectories showing that recommended procedures can become an execution burden. Relevance rankings overlook more useful candidates. Within the evaluated candidate sets, reranking by support for required operations raises first-choice pass rates by 4.35--5.80 percentage points across the three configurations. We derive 17 authoring practices linking executable procedures to recovery, preservation of task requirements, and checks on final artifacts. Stage Plan and Dependency DAG outperform use order alone, with DAG's additional benefits concentrated in tasks supplied with five or six Skills. These findings guide developers to assess usable operation support, allow procedure adaptation while preserving task requirements, and make artifact dependencies explicit when organizing Skills.

Explore related subjects

cs.AIcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yu Cheng, Dehai Zhao, Zhongxin Liu, Qing Huang, Zhenchang Xing, Xiaoxue Ren. 2026-10-08. An Empirical Study of Agent Skills' Downstream Utility. https://arxiv.org/abs/2610.08875

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulation

The integration of Large Language Models (LLMs) into social simulation has generated enthusiasm but also raises methodological and epistemological challenges. This critical review examines LLMs as cognitive or decision-making components of simulated agents from a computational social science perspective. We argue that their behavioral and epistemic limitations can become systemic vulnerabilities when scaled to multi-agent societies. We map the evolving landscape of LLM-driven platforms, from narrative sandboxes to population-scale simulations, and develop a framework for analyzing the Micro-to-Macro Validity Gap: the propagation, amplification, or contraction of micro-level limitations-including hallucinations, stochastic inconsistency, representational biases, and alignment effects-into agent-level distortions (e.g., persona convergence) and macro-level risks (e.g., artificial consensus and behavioral homogenization). We also highlight evaluative vulnerabilities such as the Fluency Fallacy and automation bias. We identify contexts where LLM-based agents offer operational value, including serious games, participatory environments, and exploratory modeling, distinguishing these uses from confirmatory research with unconstrained models and precise social forecasting. Finally, we examine theory-driven hybrid architectures embedding LLMs in explicit, mechanistic, and reproducible Agent-Based Modeling (ABM) frameworks. While promising, these architectures do not guarantee epistemic control: validity requires multi-level evaluation of environmental dynamics, individual behavior, cross-level interactions, and aggregate outcomes. They also remain vulnerable to physics washing.

cs.AI↗
arXiv2026

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

On-device LLM inference is increasingly important for edge platforms but remains challenging due to high computational and memory demands and tight resource constraints. Modern applications often involve long prompts. Although compact models reduce linear-layer costs through weight compression, attention remains largely unoptimized and becomes the dominant bottleneck, especially during prefilling. Existing attention pruning methods primarily target highly parallel hardware (e.g., GPUs) and long context windows, limiting their suitability for edge scenarios. We present DeltaLLM, a training-free LLM framework based on algorithm-hardware-dataflow co-design to accelerate prefilling attention. DeltaLLM introduces a causal-attention-aware K-only delta formulation that enables Delta-Key reuse across query heads under Grouped-Query Attention (GQA). A context-aware hybrid strategy preserves exact attention near the diagonal while applying delta approximation to nonlocal regions. For score-value multiplication, we propose a vectorized structured sparse online softmax (VSSOS) that converts fine-grained spatial score sparsity into hardware-friendly column-level skipping. A specialized vector-processor architecture and unified sparse/dense attention dataflow exploit the resulting sparsity. Evaluated on BitNet-b1.58-2B-4T and LLaMA3.2-1B-Instruct, DeltaLLM achieves up to 60% Delta-Key sparsity and 63% structured attention-score sparsity with negligible average accuracy loss. On the KV260 FPGA board, it delivers a 1.62x speedup over TeLLMe-v2 while reducing DSP and BRAM usage. These results show that DeltaLLM supports efficient edge LLM deployment and integration with existing pipelines without fine-tuning.

cs.AI↗
arXiv2026

The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation

Benchmarking the capabilities of AI systems, including Large Language Models (LLMs) and Vision Models, typically ignores the impact of uncertainty in the underlying ground truth answers from experts. This ambiguity is not just limited to human preferences, but is also consequential even in safety critical domains such as medicine where uncertainty is pervasive. In this paper, we introduce a probabilistic paradigm to theoretically explain how high certainty in ground truth answers is almost always necessary for even an expert to achieve high scores. In contrast, in datasets with high variation in ground truth answers, the expert performance drops substantially resulting in almost no difference between a random labeller and an expert. This characteristic also manifests when comparing models, where uncertainty obfuscates differences between poor and high performing models. Therefore, ignoring uncertainty in ground truth evaluation data can result in the misleading conclusion that a non-expert has similar performance to that of an expert. Given ground truth answer variability and commonly used F1 and accuracy metrics, we bring forth a simple and explainable probabilistic coin toss model to estimate expected accuracyand expected F1 and compare the estimated scores with those that experts (human or model) achieve across multiple domains -- including radiology and models such as GPT-5 and Gemini-3. The results lead to the recommendation that stratification by the probability of the ground truth answer becomes critical when expert performance is relatively low. Under stratified evaluation, performance comparison becomes more reliable in high certainty bins, mitigating the effect of the key confounding factor --- uncertainty.

cs.AI↗