Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

Chayapatr Archiwaranguprok; Krystal Kirkland; Rachel Poonsiriwong; Stanley Huang; Constanze Albrecht; Jackson Trager; Emillie de Keulenaar; Sheer Karny; Peter Baxter; Nikos Bellos; Andre Luiz Freire de Carvalho Kato; Yuning Liu; Valdemar Danry; Auren Liu; Jennifer Pfister; Pattie Maes; Nathanael J. Fast; Ravi Iyer; Juliana Schroeder; Pat Pataranutaporn

arXiv · 2610.20690

Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

View original source ↗Read source document ↗Export metadata ↓

Abstract

The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.

Explore related subjects

cs.HC cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn. 2026-10-14. Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans. https://arxiv.org/abs/2610.20690

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FlexMind: Supporting Deeper Creative Thinking with LLMs

Expert ideation requires more than generating ideas. People must articulate trade-offs, develop mitigations, and compare parallel directions, yet this reasoning process is cognitively demanding. Existing ideation tools fall short by either focusing only on expanding or organizing the idea space, or by offering evaluations without supporting mitigation generation and continued reasoning over newly introduced trade-offs. We present FlexMind, an AI-augmented system that proactively scaffolds multi-threaded trade-off-mitigation chains within a structured workspace. FlexMind helps users surface concrete trade-offs, generate actionable mitigations, and navigate and expand multiple idea branches. In a controlled study (n=20), FlexMind supported deeper exploration without sacrificing breadth and produced higher-quality ideas, as judged by domain experts, compared to a baseline. A case study with six experts (n=6) further reveals how FlexMind supports idea analysis while helping uncover new solutions. These results inform implications for enhancing human-AI co-creation.

cs.HC↗

arXiv2026

ZORO: Active Controls to Support Rule Following in Vibe Coding

Rules files (e.g., AGENTS$.$md, CLAUDE$.$md) are the primary mechanism for human-agent alignment when developers vibe code. However, they remain passive: it is not immediately apparent when rules are being used or followed, or how to improve them. To transform rules from passive text into active controls, we introduce ZORO, an interactive interface that integrates directly with a coding agent and anchors rules to every step of the coding process. After an agent generates an initial plan, ZORO enriches the plan with rules, enforces the rules during implementation by requiring the agent prove that each rule was followed, and allows users to provide in-situ feedback when they are unsatisfied with a rule application to evolve the ruleset. A technical evaluation shows that coding agents follow rules more with ZORO than without. A user study demonstrates a change in people's behavior and cognitive strategies when rules are at the forefront of vibe coding. We discuss how making rules active in agentic systems unlocks broader opportunities for human-agent alignment in coding settings and beyond.

cs.HC↗

arXiv2026

Calibrating LLM Judges for Human and AI Conversations

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

cs.HC↗