Insights into the effects of indicators on knowledge production in Astronomy

Abstract

Nine interviews were conducted with astronomers from Leiden University, and a document analysis was performed on relevant institutional (self-) evaluation documents, annual reports, and CVs of the interviewees. The aim was to perform a qualitative study about the relationship between the research behaviour of astronomers and how their science is being evaluated. This study encompassed the funding and publication system as well as the indicators used to measure the scientific output, its quality and the research performance. This report sheds light on how astronomers define high-quality research and how they think that creating knowledge of value is encouraged or hampered by the evaluation processes. We found that astronomers are realists who define scientific quality on the basis of "truth" and are driven by curiosity. These two factors make up their intrinsic values and motivation to perform Astronomy. Publication pressure, arising from the requirements of "the system", creates an extrinsic motivation to perform. This results in premature publications, low readability and replicability, risk aversion and a focus on quantity rather than quality. Hence, indicators do not merely represent quality, but also co-constitute what counts as good research. While we observe such constitutive effects of indicator use on research behaviour and content, we do not see that the astronomer's intrinsic values are co-constituted. This gives rise to a discrepancy between what is being measured by indicators and what astronomers define as scientific quality; the so-called "evaluation gap". Findings on constitutive effects and the evaluation gap in Astronomy lays out the conceptual groundwork for further empirical research and for policy advice on alternative evaluation practices and innovative indicators with the aim of bridging the "evaluation gap".