What FLARE's Published Numbers Actually Measure
We audit the scoring pipeline of FLARE, the financial task suite behind PIXIU and FinBen, and find that in a decisive fraction of cells the score is a property of the answer-extraction rule, not the model. The clearest evidence is already published: of the 34 released Headlines scores, nine equal to four decimals what the benchmark's own code returns when no output is byte-equal to Yes, and only four clear the per-stratum majority floor. FLARE credits the first entry of the dataset card's choices list found anywhere in a generation; the lm-evaluation-harness it forks takes the first match by position; on identical generations the two differ by up to 0.4216 accuracy, and no leaderboard states which rule produced its numbers. An unspecified rule leaves the score partially identified, and because the coarsening is multiplicity rather than missingness the interval's width is counted rather than bounded. Across a 204-cell factorial and a 46-cell diagnostic, 10 cells are certified invariant to the rule, 4 only because almost nothing parses, so certification is paired with coverage. On Financial PhraseBank four of seven models fall below the 0.5948 majority floor under one shipped rule and clear it under the other; on three of four substring-scored tasks no open-weight model we test is distinguishable from a degenerate predictor. Reading a FLARE score needs three quantities it does not report -- ambiguity rate, coverage, floor -- and floorbench computes all three from generations already produced.