Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation

Abstract

An LLM judge assesses agent steps from text packets. A renderer converts recorded runs into these packets, while software checkers test records and packets against requirements. We ask what evidence shows a packet is what the evaluation requires and propose four input-assurance obligations: required content survives rendering, checkers cover their requirements, a failed check stops later use and the judge reads the bytes last checked. In one pipeline with 30 trajectories, matching hashes, passing checkers and successful staging (copying packets for the judge) coexisted with violations of all four obligations, the last two in integration tests. The renderer omitted every structured tool call, leaving 53 of 60 judged steps blank despite matching hashes. For one judge configuration, later context raised error flags by 13 more on error than on correct steps. Restoring every call reduced this contrast to 2, a reduction of 11 (0.37 per trajectory, 95% cluster-bootstrap interval 0.17 to 0.53). In packets ending at the judged step, that configuration abstained at least once on every blank step, while the other four sometimes abstained but also flagged blank error and correct steps. Both separately written checkers accepted an unmatched tool result. In integration tests, a stub replacing the judge read a failed build's packet and a copy changed after staging. Three added controls stopped 18 of 19 constructed fault cases before the stub read any packet. The remaining stale script never ran. We turn the obligations into four tests and plan to evaluate their transfer to independent pipelines.