How to Build a Benchmark That Actually Measures Taste

Xiaofan Wu; Xi Zeng; Miaoxia Chen; Peishan Chen; Shuyan Li; Chen Wang; Xiaohan Wang; Wanling Xu; Jiyun Yao; Hanyong Zhong; Jiahao Zhu

arXiv · 2610.14707

How to Build a Benchmark That Actually Measures Taste

Xiaofan Wu·Xi Zeng·Miaoxia Chen·Peishan Chen·Shuyan Li·Chen Wang·Xiaohan Wang·Wanling Xu·Jiyun Yao·Hanyong Zhong·Jiahao Zhu

View original source ↗Read source document ↗Export metadata ↓

Abstract

Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benchmark across visual, brand, book, and tech items under a frozen rubric with expert-only gold (panel n>=3; percent agreement >=0.60 primary gate; weak engagement labels train/analysis-only), together with a control protocol that separates craft judgment from test-gaming. Applying the protocol to a Round-1 pilot (primary_human_expert; three biological human experts; n=250 pairs) surfaces and quantifies three validity threats: two construction artifacts that inflate the trivial-baseline layer, and a difficulty ceiling that inflates every score. (1) Position: the preferred item is slot A with probability 0.956, so an always_item_a policy scores 0.956 (fair chance 0.5). (2) Lexical leakage: a filename token cue scores 0.912. (3) Difficulty ceiling: craft domains show near-ceiling human agreement (book/brand/tech 1.0), and even under A/B randomization (eval_assignment_seed7; gold_side_A ~0.47) all five VLM cards reach 1.0 on craft, with only visual separating (~0.84-0.91; a post-hoc hard subset, n=107, leaves craft at 1.0). The protocol (expert-only gold, a weak-label firewall, side randomization at construction, filename scrubbing, and hard filters) makes each artifact measurable and, for the two construction artifacts, correctable; we release a reference eval harness and a side-randomization tool, and show the tool removes the position skew (0.916 to 0.495/0.505). The result is a calibrated view of what the benchmark does and does not yet measure: position and leakage are fixable, whereas the craft-domain ceiling is a pair-difficulty limit that Round-2 must address. We do not evaluate engagement, era/circle conditioning, or calibration here, and we claim no general taste SOTA.

Explore related subjects

cs.AI cs.CL cs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Xiaofan Wu, Xi Zeng, Miaoxia Chen, Peishan Chen, Shuyan Li, Chen Wang, Xiaohan Wang, Wanling Xu, Jiyun Yao, Hanyong Zhong, Jiahao Zhu. 2026-10-14. How to Build a Benchmark That Actually Measures Taste. https://arxiv.org/abs/2610.14707

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

RoSeViT: Role-Separated Vision Transformers for ARC Visual Reasoning

ARC tasks can be formulated as visual transformation problems, where a model must infer a high-level transformation rule from a small set of demonstrations and apply it consistently across a grid. While dense Vision Transformers (ViTs) perform well in this setting, they entangle local visual state and global transformation logic within a shared token representation, which can hinder consistent reasoning. In this paper, we introduce RoSeViT, a role-separated Vision Transformer that explicitly decouples high-level instruction inference from visual workspace computation. RoSeViT uses controller tokens to aggregate transformation-level information and workspace tokens to represent the visual grid, with a structured attention pass in which nonlocal workspace interactions are routed through the controller. This design is motivated by our formal analysis showing that enforcing a shared instruction representation can reduce the effective complexity of visual transformation modeling. RoSeViT outperforms dense ViT baselines under the standard VARC training and evaluation protocol for ARC. In a controlled backbone replacement setting, it improves ARC-1 accuracy from 54.5 to 56.6, and with recurrent refinement reaches 58.8. When integrated into the full system, RoSeViT-Ensemble improves performance from 60.4 to 62.6 on ARC-1. These results demonstrate that explicitly separating instruction and workspace representations leads to more effective visual reasoning on ARC tasks. GitHub repository: https://github.com/lz7fd/RoSeViT.

cs.AI↗

arXiv2026

Learning to Model Optimization Problems from Solver Outcomes

Automating optimization modeling with LLMs is a promising path toward scalable decision intelligence, but existing approaches either rely on agentic pipelines built on closed-source LLMs with high inference latency or fine-tune smaller LLMs using costly process supervision that often overfits to a single solver API. We study how reinforcement learning from inexpensive outcome labels can improve optimization modeling and support reuse across heterogeneous solver environments. To this end, we propose Optimization Modeling from Solver Outcomes (OMSO), which uses reinforcement learning (RL) with a mathematical programming solver as a deterministic, interactive environment. Given a natural-language problem and a target solver, OMSO generates solver-specific code, executes it in a sandboxed harness, and converts execution outcomes into scalar rewards in a closed-loop generate-execute-feedback-update process. This RL formulation learns optimization modeling without process-level supervision and supports cross-solver generalization by switching the solver environment rather than reconstructing solver-specific datasets. Experiments on NL4OPT, MAMO, IndustryOR, and OptiBench across Gurobi, OR-Tools, and COPT show that OMSO matches or outperforms process-supervised SFT, supports zero-shot training-free transfer, direct RL training, and SFT-cold-started RL, and improves modeling reasoning and semantic correctness beyond solver API usage.

cs.AI↗

arXiv2026

Forget Attention: Importance-Aware Attention Is All You Need

Attention retrieves content precisely, but its scores depend only on query--key products, leaving order and recency to positional encodings. Conversely, state-space models (SSMs) such as Mamba-3 track order and recency through input-dependent state transitions that decay and rotate, yet compressing context into a fixed-size state hampers precise retrieval. Hybrids like Samba and Hymba exploit this complementarity only across layers or heads, so SSM transitions never shape attention scores directly. The closest prior work, the Forgetting Transformer (FoX), adds an input-dependent forget gate to the scores but reweights attention by recency alone. We propose SSM-Informed Softmax Attention (SISA), which builds Mamba-3-style state transitions, rotation included, directly into the scores alongside FoX's gate. Each logit receives the weight such an SSM would give that token pair: a decayed match between rotated SSM projections. This rotation already encodes relative position, so SISA drops RoPE; restoring it raises loss. SISA outperforms parameter-matched Transformer and FoX in language modeling and length extrapolation. Among the attention models we compare, hybrids included, only SISA benefits from context beyond the training window. Building SSM transitions into the scores yields stronger attention, and our kernels make it faster and leaner than baselines on their official or standard kernels.

cs.AI↗