Cost-effective testing of a product in product line requires employing systematic and automated approaches often based on models (such as feature models). With this in mind and driven by needs of our industrial collaborations, we developed a Product Line Test Management Framework (PL-TMF) focusing on managing and generating test cases product lines.
We focused on three key problems: selecting a set of relevant test cases for a new product from a large set of test cases available for the product line, minimizing the selected test cases to obtain an optimal set of test cases meeting all the cost/effectiveness criteria, and generating test cases for the new functionalities based on behavioral models. For each of the problem, we developed a methodology with the tool support.
First, based on the expertise and discussions with test engineers in Cisco, we developed a Feature Model for Testing (FM_T) to capture the commonalities and variabilities of the product line using a commercial tool namely Pure::Variants. Second, a Component Family Model for Testing (CFM_T) was automatically developed to capture the overall test case structure in the Test Cases repository using a tool we developed called Import Plugin and Transformation (IPT). By linking CFM_T and FM_T with restrictions (also built automatically by IPT), test engineers only requires to select a set of features for a product through Pure::Variants and a set of relevant of test cases will be obtained automatically. Last, the obtained test suite can be put into a test execution tool (developed by Cisco) for testing the product.
While selecting a set of relevant test cases for a product, there can still be redundant test cases existing in the selected test suite, which require minimization for reducing the cost of testing (e.g., execution time). Such minimization needs to preserve high effectiveness as compared with the original test suite, which can be considered as a multi-objective optimization problem and solved by search algorithms. Based on that, we proposed an automated approach by defining five cost/effectiveness measures (e.g., fault detection capability) and formulating them as a fitness function.
The proposed cost/effectiveness measures and fitness function were evaluated in conjunction with various search algorithms using the Cisco industrial case study, five case studies from SPLOT and 500 artificial problem of varying complexity.
It is well known that major effort to apply MBT in practice is to develop models for SUT systems in order to generate test cases. In our work (see related publications), four types of variability for a product line were captured using standard UML class diagrams, UML state machine diagrams, aspect class diagrams and aspect state machines, which were developed by a commercial tool called IBM Rational Software Architect (RSA). All these models are stored in a repository of behavioral Models and used for test case generation after configuring them for each new product. However, using such models for generation, test engineers are required to be familiar with concepts of all the behavioral models (e.g., UML class diagram). To ease the adoption of MBT in practice, we proposed a test generation methodology using FM and CFM to shield test engineers from all the above modeling expertise.
More specifically, we first use the FM_T to capture the commonalities and variabilities of a product line. Second, a Component Family Model for Behaviors (CFM_B) is developed to associate the behavioral models in the repository from system information (e.g., API information and system states). Notice CFM_B can be built automatically using a tool we developed namely Import Plugin and Transformation for Behaviors (IPTB). By linking CFM_B and FM_T via restrictions, test engineers are only required to perform selection and configure related parameters in FM_T and all relevant behavioral models will be selected and configured automatically. Finally, the configured models will be given as in input to a tool called TRansformation-based tool for Uml-baSed Testing (TRUST) for generating executable test cases used to test the SUT system through the test execution tool.
The test generation methodology was also applied to the Saturn product line developed by Cisco and its four products. The results showed the complexity of configuration can be reduced significantly (on average 66.7% modeling effort is reduced.