Probably Approximately Correct Guarantees for Data-Driven Reachability Analysis: A Theoretical and Empirical Comparison
Reachability analysis evaluates system safety by identifying the set in which states may evolve over a time horizon. Data-driven reachability analysis estimates reachable sets and derives probabilistic guarantees directly from data. Several popular techniques for validating reachable sets---conformal prediction, scenario optimization, and the holdout method---admit similar Probably Approximately Correct (PAC) guarantees. We establish a formal connection between these PAC bounds and present an empirical case study on reachable sets to illustrate the trade-offs associated with these methods. We argue that despite the formal relationship between these techniques, subtle differences arise in both the interpretation of guarantees and the parameterization. We conclude with practical advice on the usage of these methods.