An Empirical Evaluation of Mutation and Crossover Operators for Multi-Objective Uncertainty-Wise Test Minimization

Shaukat Ali1,Yan Li1, Tao Yue1,2, Man Zhang1

1Simula Research Laboratory, Oslo, Norway; 2Department of Informatics, University of Oslo, Oslo, Norway;
Contact: {shaukat,adeng,tao, manzhang}


Multi-objective uncertainty-wise test case minimization focuses on selecting a minimum number of test cases to execute out of all the available ones while maximizing effectiveness (e.g., coverage), minimizing cost (e.g., time to execute test cases), and at the same time optimizing uncertainty-related objectives. In our previous work, we developed four such uncertainty-wise test case minimization strategies relying on Uncertainty Theory and multi-objective search (NSGA-II with default settings) that were evaluated with one real Cyber-Physical System (CPS) with inherent uncertainty. However, a fundamental question to answer is whether these default settings of NSGA-II are good enough to provide optimized solutions. In this direction, we report one of the preliminary empirical evaluations, where we performed an experiment with three different mutation operators and three crossover operators, i.e., in total nine combinations with NSGA-II for the four uncertainty-wise test case minimization strategies using a real CPS case study. The results showed that Blend Alpha crossover operator with polynomial mutation operator permits NSGA-II achieving the best performance in terms of solving our uncertainty-wise test minimization problems.

For implementation:

As mentioned in the paper, all the selected search algorithms and quality indicators are implemented based on jMetal [1].

For the experiment data: We provide the industrial case (Bandy)experiment data, including the data for objective function values,corresponding solutions and time taken by running NSGA-II of 9 combinations with different crossover and mutation operators.

Data related with the experiment

Download The data related with the industrial case (Bandy)