Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2311.06180

Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning

Tong Wan·Zhongzhou Chen
View original source ↗Read source document ↗Export metadata ↓

Abstract

Instructor's feedback plays a critical role in students' development of conceptual understanding and reasoning skills. However, grading student written responses and providing personalized feedback can take a substantial amount of time. In this study, we explore using GPT-3.5 to write feedback to student written responses to conceptual questions with prompt engineering and few-shot learning techniques. In stage one, we used a small portion (n=20) of the student responses on one conceptual question to iteratively train GPT. Four of the responses paired with human-written feedback were included in the prompt as examples for GPT. We tasked GPT to generate feedback to the other 16 responses, and we refined the prompt after several iterations. In stage two, we gave four student researchers the 16 responses as well as two versions of feedback, one written by the authors and the other by GPT. Students were asked to rate the correctness and usefulness of each feedback, and to indicate which one was generated by GPT. The results showed that students tended to rate the feedback by human and GPT equally on correctness, but they all rated the feedback by GPT as more useful. Additionally, the successful rates of identifying GPT's feedback were low, ranging from 0.1 to 0.6. In stage three, we tasked GPT to generate feedback to the rest of the student responses (n=65). The feedback was rated by four instructors based on the extent of modification needed if they were to give the feedback to students. All the instructors rated approximately 70% of the feedback statements needing only minor or no modification. This study demonstrated the feasibility of using Generative AI as an assistant to generating feedback for student written responses with only a relatively small number of examples. An AI assistance can be one of the solutions to substantially reduce time spent on grading student written responses.

Explore related subjects

physics.ed-ph

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tong Wan, Zhongzhou Chen. 2024-02-27. Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning. https://doi.org/10.1103/physrevphyseducres.20.010152

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Data-driven modeling in the introductory physics laboratory: Scaling analysis and data collapse in the specific heat of water experiment

In introductory physics laboratories, a central instructional goal is to help students construct and evaluate mathematical models from empirical data rather than applying given formulas. We present a data-driven redesign of the classic specific heat of water experiment that emphasizes scaling analysis and data collapse as tools for model construction. The activity combines structured experimental and analytical guidance with instructor-mediated questioning, while thermodynamic theory is deliberately postponed. Students collect temperature-time data under various experimental conditions, producing multiple data sets that initially appear unrelated. Through successive rescaling, students reduce the dimensionality of the variable space and achieve data collapse onto a single master curve, from which they formulate an empirical model relating energy input, mass, and temperature change. The analysis highlights a limitation of multiplicative scaling: the additive contribution of the calorimeter cannot be eliminated, leading to a structural non-identifiability of the subsystem contributions. To clarify the domain of validity of the model, a thermodynamic description is introduced a posteriori as a boundary-setting framework for interpreting the empirical model. In this sense, the central contribution of this work is to use data-driven modeling both to construct models and to reveal their intrinsic limitations. The experiment provides an accessible example of how scaling, data collapse, and theoretical reasoning can be integrated in an introductory laboratory.

physics.ed-ph↗
arXiv2026

Missing Data on Physics Exams: Demographic Patterns, Course-Level Predictions, and Implications for Equity

In a previous quantitative retrospective study we showed that different demographic groups of students leave different numbers of problems blank on physics exams, leading to inequities in course outcomes. In that work we argued that there were good reasons to treat these blanks as missing data, rather than indicators of a lack of understanding. In this paper, we refine this analysis and show more detailed breakdowns of uncollected test item responses by race/ethnicity and first generation college student status, coming to the same conclusion: test item responses are uncollected for students with different ethnic and racial backgrounds at different rates, and these patterns are not exclusive to low-performing students. We also correct an error from our previous work, finding here that there is no significant gender difference in uncollected test item responses. Finally, we provide a more robust analysis of course level data illustrating that blanks are a variable controlled at the course level rather than the student level, providing more evidence for the use of a course deficit model (rather than a student deficit model) when examining equity disparities, and also suggesting that there are plausible means for instructors to minimize uncollected test item responses, and therefore reduce or even eliminate the bias associated with this missing data. We provide a couple suggestions for faculty who want to minimize the impacts of blanks.

physics.ed-ph↗
arXiv2026

A Framework for Characterizing Learning Contributions Across the Initial Achievement Spectrum

Conceptual assessments are widely used in physics education research to evaluate changes in student understanding, yet class average measures can obscure how those changes are distributed across students with different levels of initial achievement. We introduce a Learning Contribution Framework that characterizes this distribution through the Learning Contribution Curve (LCC) and Learning Contribution Profile (LCP). For a general contribution measure G, the LCC represents cumulative contribution across students ranked by initial achievement, whereas the LCP describes the local mean contribution relative to the population mean of the individual G values. We apply the framework to the individual Hake normalized gain and develop a statistical model linking LCC and LCP behavior to the joint structure of pretest and posttest scores. Under the central assumption that the conditional mean of the subsequent score is linear in the initial score, we identify a score structure parameter $β$ and a critical score structure parameter $β_c$. Whether $β$ is greater than, less than, or equal to $β_c$ determines whether the expected LCP increases, decreases, or remains constant across initial achievement. Given the pretest distribution, the model further yields analytical predictions for the complete LCP and LCC that closely reproduce simulation results. Application to classroom concept-assessment data illustrates how the framework reveals local and cumulative patterns of learning contribution that are not evident from an overall class average revealed by the traditional Hake's normalized gain.

physics.ed-ph↗