Prompt perturbation and fraction facilitation sometimes strengthen Large Language Model scores

Mike Thelwall

Home/Archive/arXiv

arXiv · 2512.01330

Prompt perturbation and fraction facilitation sometimes strengthen Large Language Model scores

Mike Thelwall

View original source ↗Read source document ↗Export metadata ↓

Abstract

Large Language Models (LLMs) can be tasked with scoring texts according to pre-defined criteria and on a defined scale, but there is no recognised optimal prompting strategy for this. This article focuses on the task of LLMs scoring journal articles for research quality on a four-point scale, testing how user prompt design can enhance this ability. Based primarily on 1.7 million Gemma3 27b queries for 2780 health and life science articles with 58 similar prompts, the results show that improvements can be obtained by (a) testing semantically equivalent prompt variations, (b) averaging scores from semantically equivalent prompts, (c) specifying that fractional scores are allowed, and possibly also (d) not drawing attention to the input being partial. Whilst (a) and (d) suggests that models can be sensitive to how a task is phrased, (b) and (c) suggest that strategies to leverage more of the model's knowledge are helpful, such as by perturbing prompts and facilitating fractions. Perhaps counterintuitively, encouraging incorrect answers (fractions for this task) releases useful information about the model's certainty about its answers. Mixing semantically equivalent prompts also reduces the chance of getting no score for an input. Additional testing showed that the best prompts vary between LLMs, however, and were almost the opposite for ChatGPT 4o-mini, weakly aligned for Llama4 Scout and Magistral, and made little difference to Qwen3 32b and DeepSeek R1 32b. Overall, whilst there is no single best prompt, a good strategy for all models was to average the scores from a range of different semantically equivalent or similar prompts.

Explore related subjects

cs.DL

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Mike Thelwall. 2025-12-01. Prompt perturbation and fraction facilitation sometimes strengthen Large Language Model scores. https://arxiv.org/abs/2512.01330

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research

Automated scholarly paper review (ASPR) has entered the coexistence phase with traditional peer review, where artificial intelligence (AI) systems are increasingly incorporated into real-world manuscript evaluation. In parallel, research on automated and AI-assisted peer review has proliferated. Despite this momentum, empirical progress remains constrained by several critical limitations in existing datasets. While reviewers routinely evaluate figures, tables, and complex layouts to assess scientific claims, most existing datasets remain overwhelmingly text-centric. This bias is reinforced by a narrow focus on data from computer science publications. Furthermore, existing datasets rarely preserve precise alignment between review comments and specific manuscript versions, obscuring the iterative relationship between peer review and manuscript evolution. In response, we introduce FMMD, a multimodal and multidisciplinary open peer review dataset curated from F1000Research. The dataset addresses the current limitations by integrating manuscript-level visual and structural data with version-specific reviewer reports and editorial decisions. By explicitly aligning review comments with the exact article version under review, FMMD enables granular analysis of the peer review lifecycle. Importantly, its coverage of F1000Research extends ASPR research beyond its traditional focus on computer science to a diverse range of scientific disciplines. FMMD supports a range of research tasks, including visual-semantic consistency classification, figure-related review comment generation, and editorial decision prediction based on multimodal manuscript inputs, thereby providing a comprehensive empirical resource for developing and evaluating multimodal ASPR systems and advancing peer review research.

cs.DL↗

arXiv2026

GitScholar: A Dataset for Predicting AI Research Impact from GitHub Engagement

With the rapid pace of AI research and the hundreds of daily new publications, staying up-to-date with the latest developments has become increasingly difficult. For researchers, quickly identifying impactful work is essential, yet manually reviewing each new publication is impractical. Automated impact prediction methods help address this challenge, usually by combining various information sources available, such as a paper's content or citation history. In this work, we propose using GitHub engagement as an additional source and demonstrate that it provides both a timely and accurate signal. To this end, we introduce GitScholar, a novel dataset that links GitHub activity from 444,000 repositories to over 558,000 AI arXiv papers. Our experiments show that GitHub reactions improve early prediction precision by up to 12% over a strong academic baseline. Additionally, we find that GitHub signal offers near-complete coverage of high-impact AI papers, and consistently correlates with future academic success. GitScholar is publicly available at https://huggingface.co/datasets/huawei-csl/GitScholar.

cs.DL↗

arXiv2026

The illusion of neutrality in metric-based research evaluation

Venue prestige and citation counts are two widely used, albeit imperfect, signals of research quality. When the two signals conflict, evaluators must decide how much weight to assign each. Yet, it remains unknown how researchers across disciplines trade off these two signals when evaluating research outcome. To fill this gap, we surveyed 869 researchers using paired choices between hypothetical departmental hiring rules that assigned different weights to venue prestige and citation counts, asking which would produce better science. Among 795 respondents whose choices were largely internally consistent, choices placed nearly equal aggregate weight on the two signals. This apparent balance concealed substantial individual heterogeneity: nearly two in five respondents occupied the most venue-heavy or citation-heavy intervals. Moreover, respondents on average chose more citation-heavy rules than they believed their departments used in hiring. By revealing the subjective judgments that arise when research indicators conflict, our findings reinforce calls for greater caution when quantitative indicators are used to evaluate research.

cs.DL↗