Uncertainty-Wise and Time-Aware Test Case Prioritization with Multi-Objective Search

Shaukat Ali1,Yan Li1, Tao Yue1,2, Man Zhang1

1Simula Research Laboratory, Oslo, Norway; 2Department of Informatics, University of Oslo, Oslo, Norway;
Contact: {shaukat,adeng,tao, manzhang}@simula.no

Abstract

Complex systems (e.g., Cyber-Physical Systems) that interact with the real world, behave in an unstipulated manner while operating in uncertain environments. Testing such systems in uncertainty is a big challenge. Devising uncertainty-wise testing solutions can be considered as a mandate for dealing with this challenge. Though uncertainty-wise testing is gaining attention in the last few years, industry-strengthening solutions are still missing. In this paper, we propose an uncertainty-wise, search-based, multi-objective test case prioritization approach, with a fitness function defined based on four cost-effectiveness measures: one subjective and one objective uncertainty measures, execution time, and transition coverage. We evaluated the well-known multi-objective search algorithm NSGA-II by comparing it with Greedy and Random Search (RS), with a real industrial case study. In addition, we created 72 additional simulated problems of varying complexity based on the real case study. Results show that NSGA-II achieved significantly better performance than RS and Greedy for both the real industrial case study and the simulated problems. On average. NSGA-II improved prioritization by 18% and 22% as compared to RS and Greedy respectively

For implementation:

As mentioned in the paper, all the selected search algorithms and quality indicators are implemented based on jMetal [1].

For the experiment data: Two types of data are made available: 1) We provide the industrial case (Bandy)experiment data, including the data for objective function values,corresponding solutions and time taken by running each algorithm. 2) We provide simulated problems experiment data, including the current test case repository of GS with 2085 test cases to generate these simulated problems and the data for objective function values,corresponding solutions and time taken by running each algorithm.

Data related with the experiment

Download The data related with the real world case (Bandy)

Download The data related with simulated problems(72 problems)