Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2606.05405

Agents' Last Exam

Yiyou Sun·Xinyang Han·Weichen Zhang·Yuanbo Pang·Tianyu Wang·Yuhan Cao·Yixiao Huang·Chris Duroiu·Haoyun Zhang·Jeffrey Lin·Weishu Zhang·Tyler Zeng·Ying Yan·Bo Liu·Hanson Wen·Mingyang Xu·Xiaoyuan Liu·Zimeng Chen·Weiyan Shi·Amanda Dsouza·Vincent Sunn Chen·Patrick Bryant·Carl Boettiger·Yamini Rangan·Bradley Rothenberg·Kyle Steinfeld·Arvind Rao·Tapio Schneider·Georgios Yannakakis·Laure Zanna·Kaan Ozbay·Ida Sim·Tarek Zohdi·George Em Karniadakis·Jack Gallant·Teresa Head-Gordon·Yushan Li·Wenxi Deng·Tao Sun·Huiqi Wang·Zhun Wang·Justin Xu·Chris Yuhao Liu·Yafei Cheng·Rongwang Hu·Aras Bacho·Shengcao Cao·Zengyi Qin·Yixiong Chen·Hengduan Fan·Hao Liu·Lin Zeng·Shashank Muralidhar Bharadwaj·Litian Gong·Yingxuan Yang·Maojia Song·Ruheng Wang·Zongzheng Zhang·Honglin Bao·Shuo Lu·Jianhong Tu·Zhonghua Wang·Zheng Zhang·Zijiao Chen·Yanqiong Jiang·Zhendong Li·Bohan Lyu·Chang Ma·Peiran Xu·Benran Zhang·Shangding Gu·Haoyue Hua·Haoyang Li·Wanzhe Liao·Chengzhi Liu·Junbo Peng·Haoran Sun·Zechen Xu·Bo Chen·Jiayi Cheng·Yi Jiang·Keying Kuang·Yuan Li·Youbang Pan·Ziyan Rao·Alexander Schubert·Yifan Shen·Vincent Siu·Xiatao Sun·Kangqi Zhang·Xiaopan Zhang·Yuchen Zhu·Ishaan Singh Chandok·Lei Ding·Jingxuan Fan·Andrew Glover·Jiaming Hu·Yiran Hu·Wenbo Huang·Zixin Jiang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

Explore related subjects

cs.AIcs.CLcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, Weishu Zhang, Tyler Zeng, Ying Yan, Bo Liu, Hanson Wen, Mingyang Xu, Xiaoyuan Liu, Zimeng Chen, Weiyan Shi, Amanda Dsouza, Vincent Sunn Chen, Patrick Bryant, Carl Boettiger, Yamini Rangan, Bradley Rothenberg, Kyle Steinfeld, Arvind Rao, Tapio Schneider, Georgios Yannakakis, Laure Zanna, Kaan Ozbay, Ida Sim, Tarek Zohdi, George Em Karniadakis, Jack Gallant, Teresa Head-Gordon, Yushan Li, Wenxi Deng, Tao Sun, Huiqi Wang, Zhun Wang, Justin Xu, Chris Yuhao Liu, Yafei Cheng, Rongwang Hu, Aras Bacho, Shengcao Cao, Zengyi Qin, Yixiong Chen, Hengduan Fan, Hao Liu, Lin Zeng, Shashank Muralidhar Bharadwaj, Litian Gong, Yingxuan Yang, Maojia Song, Ruheng Wang, Zongzheng Zhang, Honglin Bao, Shuo Lu, Jianhong Tu, Zhonghua Wang, Zheng Zhang, Zijiao Chen, Yanqiong Jiang, Zhendong Li, Bohan Lyu, Chang Ma, Peiran Xu, Benran Zhang, Shangding Gu, Haoyue Hua, Haoyang Li, Wanzhe Liao, Chengzhi Liu, Junbo Peng, Haoran Sun, Zechen Xu, Bo Chen, Jiayi Cheng, Yi Jiang, Keying Kuang, Yuan Li, Youbang Pan, Ziyan Rao, Alexander Schubert, Yifan Shen, Vincent Siu, Xiatao Sun, Kangqi Zhang, Xiaopan Zhang, Yuchen Zhu, Ishaan Singh Chandok, Lei Ding, Jingxuan Fan, Andrew Glover, Jiaming Hu, Yiran Hu, Wenbo Huang, Zixin Jiang. 2026-06-11. Agents' Last Exam. https://arxiv.org/abs/2606.05405

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

STRUCTUREDAGENT: Planning with AND/OR Trees for Long-Horizon Web Tasks

Existing LLM-based web agents struggle on complex, long-horizon tasks due to limited in-context memory, weak planning abilities, and greedy behaviors that lead to premature termination. To address these challenges, we propose \SA{}, a hierarchical planning framework that interleaves planning and execution via dynamic $\ANDOR$ trees. The framework separates structural planning from LLM-based reasoning, enabling principled error recovery through node repair, systematic exploration of alternatives via OR nodes, and modular plans that can facilitate human intervention. On WebArena (630 tasks), \SA{} achieves a $\sim$53\% success rate vs.\ $\sim$46\% for AgentOccam, and on complex multi-constraint Amazon shopping tasks, gains reach 10\% over the strongest baseline.

cs.AI↗
arXiv2026

Can Generalist Agents Automate Data Curation?

Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.

cs.AI↗
arXiv2026

Self-Organizing Agent Teams Learn to Reason Together

Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknown, useful roles and divisions of labor cannot be specified in advance; teams must learn from experience how to organize reasoning as it unfolds. Human teams routinely adapt this way, while existing AI agent teams rely on fixed protocols, explicit task decomposition, or routing. We introduce Self-Organizing Agent Teams (SAT), fixed teams of AI agents that learn reusable strategies from prior collaborations to organize roles, conversational phases, participation, and information flow. These strategies enable what we call collaborative computation: agents exchange, challenge, repair, and synthesize partial reasoning into solutions no member produced independently. In two independent settings, we learn teamwork strategies that transfer unchanged to unseen benchmarks, using only 15 mathematics and 25 graduate-level knowledge problems. Across five mathematics and physics benchmarks, self-organizing teams average 66.7% accuracy, versus 48.8% for their strongest member, 58.7% for compute-matched inference by that agent, and 59.0% for a perfect router over members' independent answers; on AIME 2026, they exceed this router by 13.4 points. Because gains vary across benchmarks, we ask when self-organizing collaboration helps. Across eight benchmarks, demonstrability (the organizational-psychology construct of whether a team can distinguish correct from incorrect reasoning) strongly tracks improvement over the strongest member (Spearman $ρ=0.90$, $p=0.005$): teams benefit most when correct reasoning can be recognized once it appears. More broadly, these results suggest that organization itself can become an agent capability: agent teams can learn how to reason together and produce solutions their members could not reach independently.

cs.AI↗