AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations in physics

Abstract

Explaining physical phenomena is central to physics learning, because students' explanations provide evidence of their conceptual understanding. Because conceptual understanding can only be inferred through language rather than observed directly, distinguishing conceptual understanding from linguistic quality represents a fundamental challenge for assessment. In this study, we examined whether AI-based scoring approaches can assess students' conceptual understanding independently of the linguistic quality of their text-based explanations in physics. We compared conceptual understanding scores generated by nine machine learning-based scoring approaches and two large language model-based scoring approaches against human-expert-assigned scores for 116 secondary-school students' explanations. Despite generally good agreement with expert-assigned scores, explanations of lower linguistic quality were systematically more likely to be underestimated, i.e. receiving lower AI-generated conceptual understanding scores than experts assigned - a language bias that emerged across every AI-based scoring approach. Higher linguistic quality showed no comparable link to overestimation. Notably, this language bias closely resembles that previously reported for physics teachers, suggesting the difficulty lies less in any particular assessor than in the nature of inferring conceptual understanding from text-based explanations itself. The stakes fall hardest on multilingual learners, whose language proficiency may be misread as weaker understanding, and grow as AI-based scoring takes on higher-stakes decisions.