Anomaly-Informed Confidence Calibration for Vision-Based Safety Prediction

Abstract

Reliable confidence estimates are important for safely deploying vision-based controllers in autonomous racing, where safety predictions must be derived from camera images, yet modern predictors become dangerously overconfident under test-time distribution shifts. A commonly assumed perception-dynamics gap holds that observation-space anomaly signals, such as autoencoder reconstruction error, should miss dynamics anomalies (e.g., actuation bias, latency) that originate outside the image. We interrogate this gap with an Anomaly-Informed Test-Time Calibration approach that, without retraining any model component, fuses a perceptual score (reconstruction error) and a disentangled dynamics score (epistemic uncertainty and control-stream statistics) from a world model to condition a lightweight temperature-scaling calibrator, aided by test-time augmentation. On a physical DonkeyCar under four real-world anomaly protocols unseen during training (darkness, blur, actuation bias, processing latency), it cuts average expected calibration error from 0.184 to 0.116, a 37% improvement over the best baseline, without modifying the base predictor. Our central finding is that in a tightly coupled closed loop, the perception-dynamics gap is not prominent. Specifically, a dynamics fault degrades the trajectory until the observation is itself out-of-distribution, so the perceptual score alone already captures most dynamics anomalies and accounts for the calibration gains. As a result, the disentangled dynamics score adds little further calibration but remains a direct, interpretable signal that aids out-of-distribution detection of dynamics and control faults.