Another reason why normalized gain should continue to be used to analyze concept inventories (and estimate learning rates)

Jairo Navarrete; Valentina Giaconi; Gonzalo Contador; Mariano Vazquez

Home/Archive/arXiv

arXiv · 2407.07730

Another reason why normalized gain should continue to be used to analyze concept inventories (and estimate learning rates)

Jairo Navarrete·Valentina Giaconi·Gonzalo Contador·Mariano Vazquez

View original source ↗Read source document ↗Export metadata ↓

Abstract

A transformation called normalized gain (ngain) has been acknowledged as one of the most common measures of knowledge growth in pretest-posttest contexts in physics education research. Recent studies in math education have shown that ngains can also be applied to assess learners' ability to acquire unfamiliar knowledge, that is, to estimate their "learning rate". This quantity is estimated from learning data through two well-known methods: computing the average ngain of the group or computing the ngain of the average learner. These two methods commonly yield different results, and prior research has concluded that the difference between them is associated with a pretest-ngains correlation. Such a correlation would suggest a bias of this learning measurement because it implies its favoring of certain subgroups of students according to their performance in pretest measurements. The present study analyzes these two estimation methods by drawing on statistical models. Our results show that the two estimation methods are equivalent when no measurement errors exist. In contrast, when there are measurement errors, the first method provides a biased estimator, whereas the second one provides an unbiased estimator. Furthermore, these measurement errors induce a spurious correlation between the pretest and ngain scores. Our results seem consistent with prior research, except they show that measurement errors in pretest and posttest scores are the source of a spurious pretest-ngain correlation. Consequently, estimating learning rates might effectively provide unbiased estimates of knowledge change that control for the effect of prior knowledge even in the presence of pretest-ngain correlations.

Explore related subjects

stat.AP

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Jairo Navarrete, Valentina Giaconi, Gonzalo Contador, Mariano Vazquez. 2024-07-10. Another reason why normalized gain should continue to be used to analyze concept inventories (and estimate learning rates). https://arxiv.org/abs/2407.07730

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Auditing Bayesian Graph Alignment: Diagnostic Comparisons and Reference Failure

Bayesian graph alignment estimates correspondence probabilities, but convergence of an alignment-score trace need not imply accurate correspondence marginals. We audit this gap on 240 new exact graph pairs from four source families, 240 larger pairs with 20-100 vertices, and a separate 60-case exact implementation check. Under an explicit edge-flip likelihood, we compare three samplers and score, marginal, indicator, categorical, and classifier-based diagnostics. Marginal disagreement improves error discrimination over score R-hat for the exact informed sampler, but its improvement for vanilla local sampling is uncertain. Assignment-based R* and short indicator panels are competitive; no diagnostic dominates across samplers and endpoints. At larger sizes, diagnostics predict subsequent marginal changes, not posterior error, and classification performance depends on the drift threshold. Disjoint-window and held-out-chain checks attenuate but preserve positive associations. Only 22 of 240 original reference sets pass an agreement screen. On forty failure-selected cases, eightfold SMC particle escalation does not resolve disagreement, whereas additional rejuvenation helps. Longer informed runs remain unstable. An elementary feasible-alignment bound demonstrates severely unrepresentative SMC and informed-chain scores in concentrated 100-vertex cases, independently of approximate reference consensus. We also exhibit common-start chains with near-zero disagreement despite exact marginal error near .967. These results support assignment-sensitive auditing while identifying limits of finite budgets, diagnostic rankings, and reference agreement as evidence of accuracy.

stat.AP↗

arXiv2026

GeoDose-CP: Graph-Local Conformal Inference for Continuous-Treatment Earth Observation

Reliable intervention-oriented uncertainty quantification from Earth observation (EO) remains challenging when continuous treatment shifts, spatial dependence, limited support, and satellite-outcome uncertainty must be addressed simultaneously. Existing causal, conformal, and spatial approaches address parts of this problem, but their direct combination does not generally recover the appropriate interventional reference law because candidate reassignment jointly alters treatment likelihood, standardized residuals, and graph-dependent residual likelihood. This study presents GeoDose-CP, a support-aware conformal framework for localized stochastic potential outcomes under continuous or mixed continuous-atomic treatment. Its central methodological contribution is a graph-local target-orbit law that jointly represents intervention-induced treatment shift, the inverse outcome-scale Jacobian, and spatial residual dependence. The framework further provides exact weighted candidate inversion, a scalable sparse approximation with explicit discrepancy accounting, and refusal under inadequate support. Evaluation used controlled known-truth experiments, MineDoseBench, treatment-density sensitivity analysis, external conformal comparators, and a multi-mine New South Wales (NSW) study. In MineDoseBench, GeoDose-CP achieved mean selective coverage of 0.9692 across 27 configurations and a minimum local q0.05 of 0.8951; exact-sparse auditing produced nine inclusion disagreements over 2,700 targets. In the NSW study, the absence of an auditable longitudinal rehabilitation treatment rendered treatment-dependent inference nonoperational rather than forcing inference through a proxy exposure.

stat.AP↗

arXiv2026

Interpreting relative utility for probabilistic predictions

At a fixed threshold, relative utility (RU) measures the net-benefit gain of a prediction model over the better of treat-all and treat-none relative to the corresponding gain under perfect outcome classification. We illustrate that RU equal to 1 therefore represents perfect outcome classification at that fixed threshold, not perfect probabilistic prediction. However, even when every predicted probability equals the true probability, observed RU can equal 0. In a simple constant-risk setting, this occurs with probability approaching 1 as the sample size increases. Consequently, the distance from observed RU to 1 should not in general be interpreted as improvement achievable by a better prediction for binary probabilities.

stat.AP↗