Guiding LLM Peer Reviewers: The Impact of Score Anchors on Review Evidence and Accuracy

Judita Preiss; Yunhan Yang

Home/Archive/arXiv

arXiv · 2609.01905

Guiding LLM Peer Reviewers: The Impact of Score Anchors on Review Evidence and Accuracy

Judita Preiss·Yunhan Yang

View original source ↗Read source document ↗Export metadata ↓

Abstract

Large language models (LLMs) are increasingly used for research quality evaluation, with prior work exploring their scoring accuracy and the plausibility of review rationales. However, less is known about whether external score guidance changes the evidence presented in the generated review as well as the final score. This study uses 98 Allied Health Professions research outputs submitted for internal REF-style assessment, with specialist human review reports and adjudicated 1-4 reference scores. No-guidance baseline reviews are compared with oracle-guided reviews, where the supplied score is set to the rounded human reference score; extracted evaluation points are used to compare human and LLM evidence use. Using this design, oracle guidance improves scoring accuracy, with score-following checks showing that models do not simply copy the supplied score. Corrected score mismatches are associated with changes in the generated review frame, showing that the score signal can steer review rationales. This effect is direction-dependent: LLM reviews cover human strength or upgrade points more reliably than human weakness or downgrade points, with the weakest alignment for expert downgrade evidence. The results show that score-guided review generation can be evaluated at the level of review evidence, as well as the final score.

Explore related subjects

cs.DL

Keep this discovery

BibTeX RIS

Judita Preiss, Yunhan Yang. 2026-09-01. Guiding LLM Peer Reviewers: The Impact of Score Anchors on Review Evidence and Accuracy. https://arxiv.org/abs/2609.01905

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.DL subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

Are Widely Known Findings Easier to Retract?

Failures of retraction are common in science. Why do they occur? And what determines whether a retraction is successful? We use data from citation records and Altmetrics to test proposed answers to these questions. LaCroix et al. employ network models to argue the social spread of information helps explain failures of retraction. One prediction is that widely known results, surprisingly, should be easier to retract, since their retraction is more relevant. Our results support this conclusion. We find highly cited papers show more significant reductions in citation after retraction and garner more attention to their retractions as they occur.

cs.DL↗

arXiv2026

Filling holes in science draws collective attention, but most higher-order holes remain unexplored

Much scientific discovery involves filling holes between ideas and arguments that unleash techno-scientific advance. Representing knowledge as high-dimensional concept embeddings, we use persistent homology to detect holes of increasing order, from gaps between disconnected ideas to higher-order cavities, and identify the research works that fill them. We find two empirical asymmetries. Researchers who fill anticipated holes are poised to draw collective attention by staging outsized novelty and foresight, indicating that bridging holes anticipates where science will converge, most strongly in empirical fields and least in formal and design fields. Yet as knowledge grows, higher-order holes explode while the fraction science fills collapses, leaving most higher-order combinations unexplored. These results call for a richer science of holes, and mark a frontier where contemporary AI might help fill the high-dimensional gaps human science opens.

cs.CY↗

arXiv2026

Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature

Tracing connections between historical texts is an important part of intertextual research, enabling scholars to reconstruct the virtual library of a writer and identify the sources influencing their creative process. These intertextual links manifest in diverse forms, ranging from direct verbatim quotations to subtle allusions and paraphrases disguised by morphological variation. Language models offer a promising path forward due to their capability of capturing semantic similarity beyond lexical overlap. However, the development of new methods for this task is held back by the scarcity of standardized benchmarks and easy-to-use datasets. We address this gap by introducing Loci Similes, a benchmark for Latin intertextuality detection comprising a curated dataset of ~176k text segments and 1,490 expert-verified parallels, including 945 labeled references from an existing dataset. Using this data, we establish baselines for retrieval and classification of intertextualities with both lexical methods and pretrained encoder language models.

cs.IR↗