When Does Permutation Instability Generalize? Independent-View Validation for Listwise LLM Reranking

Abstract

Listwise language-model rerankers often disagree across equivalent candidate permutations. Finite instability diagnostics are therefore used to motivate additional sampling, aggregation, or selective computation. But an association with a validation statistic that reuses the probe views need not isolate predictive information about unseen permutations. Shared measurements can induce classical part-whole association. We study how this affects claims that a finite-view instability score predicts unseen permutations. We derive the exact finite-view decomposition and prospectively compare zero, one, and two reused views, including a fully disjoint four-view target. The study covers two pinned 7B model families and two recommendation datasets, with controlled lists for signed offline analysis and untouched retriever lists for target-free replication. On the four controlled blocks, fully disjoint correlations are weak or heterogeneous (-0.061 to 0.281), whereas reusing both probe views yields 0.600 to 0.718; all paired contrasts are large (0.436 to 0.661) and Holm-significant. The overlap effect is positive in all four untouched-list blocks. Increasing the probe from two to four views clearly improves disjoint reliability in only one block. Moreover, the probe predicts aggregation-movement magnitude (Spearman rho = 0.142 to 0.426) but not stable signed target benefit, and 7 of 12 fixed-fraction probe-routing points are strictly dominated at measured cost. Thus, when the intended estimand is predictive information about unseen perturbation behavior, validation targets must be observation-disjoint from the probe to isolate that information; signed utility and cost-sensitive decisions remain separate questions.