Test Case Generation for Simulink Models: An Experience from the E-Bike Domain

Abstract

Cyber-physical systems development often requires engineers to search for defects in their Simulink models. Search-based software testing (SBST) is a standard technology that supports this activity. To increase practical adaption, industries need empirical evidence of the effectiveness and efficiency of (existing) SBST techniques on benchmarks from different domains and of varying complexity. To address this industrial need, this paper presents our experience assessing the effectiveness and efficiency of SBST in generating failure-revealing test cases for cyber-physical systems requirements. Our study subject is within the electric bike (e-Bike) domain and concerns the software controller of an e-Bike motor, particularly its functional, regulatory, and safety requirements. We assessed the effectiveness and efficiency of HECATE, an SBST framework for Simulink models, to analyze two software controllers. HECATE successfully identified failure-revealing test cases for 83% (30 out of 36) of our experiments. It required, on average, 1 h 17 min 26 s (min = 11 min 56 s, max = 8 h 16 min 22 s, std = 1 h 50 min 34 s) to compute the failure-revealing test cases. The developer of the e-Bike model confirmed the failures identified by HECATE. We present the lessons learned and discuss the relevance of our results for industrial applications, the state of practice improvement, and the results' generalizability.