PL-MTM

OVERVIEW

Architecture of the Zen-RUCM Framework

Cost-effective testing of a product in product line requires employing systematic and automated approaches often based on models (such as feature models). With this in mind and driven by needs of our industrial collaborations, we developed a Product Line Test Management Framework (PL-TMF) focusing on managing and generating test cases product lines.

We focused on three key problems: selecting a set of relevant test cases for a new product from a large set of test cases available for the product line, minimizing the selected test cases to obtain an optimal set of test cases meeting all the cost/effectiveness criteria, and generating test cases for the new functionalities based on behavioral models. For each of the problem, we developed a methodology with the tool support.

TEST SELECTION

TEST SELECTION

First, based on the expertise and discussions with test engineers in Cisco, we developed a Feature Model for Testing (FM_T) to capture the commonalities and variabilities of the product line using a commercial tool namely Pure::Variants. Second, a Component Family Model for Testing (CFM_T) was automatically developed to capture the overall test case structure in the Test Cases repository using a tool we developed called Import Plugin and Transformation (IPT). By linking CFM_T and FM_T with restrictions (also built automatically by IPT), test engineers only requires to select a set of features for a product through Pure::Variants and a set of relevant of test cases will be obtained automatically. Last, the obtained test suite can be put into a test execution tool (developed by Cisco) for testing the product.

TEST MINIMIZATION

TEST MINIMIZATION

While selecting a set of relevant test cases for a product, there can still be redundant test cases existing in the selected test suite, which require minimization for reducing the cost of testing (e.g., execution time). Such minimization needs to preserve high effectiveness as compared with the original test suite, which can be considered as a multi-objective optimization problem and solved by search algorithms. Based on that, we proposed an automated approach by defining five cost/effectiveness measures (e.g., fault detection capability) and formulating them as a fitness function.

The proposed cost/effectiveness measures and fitness function were evaluated in conjunction with various search algorithms using the Cisco industrial case study, five case studies from SPLOT and 500 artificial problem of varying complexity.

TEST GENERATION

TEST GENERATION

It is well known that major effort to apply MBT in practice is to develop models for SUT systems in order to generate test cases. In our work (see related publications), four types of variability for a product line were captured using standard UML class diagrams, UML state machine diagrams, aspect class diagrams and aspect state machines, which were developed by a commercial tool called IBM Rational Software Architect (RSA). All these models are stored in a repository of behavioral Models and used for test case generation after configuring them for each new product. However, using such models for generation, test engineers are required to be familiar with concepts of all the behavioral models (e.g., UML class diagram). To ease the adoption of MBT in practice, we proposed a test generation methodology using FM and CFM to shield test engineers from all the above modeling expertise.

More specifically, we first use the FM_T to capture the commonalities and variabilities of a product line. Second, a Component Family Model for Behaviors (CFM_B) is developed to associate the behavioral models in the repository from system information (e.g., API information and system states). Notice CFM_B can be built automatically using a tool we developed namely Import Plugin and Transformation for Behaviors (IPTB). By linking CFM_B and FM_T via restrictions, test engineers are only required to perform selection and configure related parameters in FM_T and all relevant behavioral models will be selected and configured automatically. Finally, the configured models will be given as in input to a tool called TRansformation-based tool for Uml-baSed Testing (TRUST) for generating executable test cases used to test the SUT system through the test execution tool.

The test generation methodology was also applied to the Saturn product line developed by Cisco and its four products. The results showed the complexity of configuration can be reduced significantly (on average 66.7% modeling effort is reduced.

RELATED PUBLICATION for PL-MTM

Journal Papers

  • S. Wang, S. Ali, A. Gotlieb, and M. Liaaen. A Systematic Test Case Selection Methodology for Product Lines: Results and Insights From an Industrial Case Study. Empirical Software Engineering (EMSE), vol 21, issue 4, pp. 1586-1622, 2016. DOI: 10.1007/s10664-014-9345-5.
  • S. Wang, S. Ali, A. Gotlieb. Cost-Effective Test Suite Minimization in Product Lines Using Search Techniques. Journal of Systems and Software (JSS) vol. 103, pp. 370-391, 2015. DOI: 10.1016/j.jss.2014.08.024.
  • S. Wang, S. Ali, A. Gotlieb, and M. Liaaen. Automated Prode duct Line Test Case Selection: Industrial Case Study and Controlled Experiment. Journal of Software and Systems Modeling (SOSYM), 2014. DOI: 10.1007/s10270-015-0462-4.
Conference Papers
  • D. Pradhan, S. Wang, S. Ali, T. Yue and M. Liaaen. CBGA-ES: A Cluster-Based Genetic Algorithm with Elitist Selection for Supporting Multi-objective Test Optimization. In The 10th IEEE International Conference on Software Testing, Verification and Validation (ICST), 2017.
  • D. Pradhan, S. Wang, S. Ali, T. Yue and M. Liaaen. STIPI: Using Search to Prioritize Test Cases based on Multi-Objectives Derived from Industrial Practice. In The 28th International Conference on Testing Software and Systems (ICTSS), pp. 172-190, 2016.
  • A. Arrieta, S. Wang, G. Sagardui and L. Etxeberria. Search-Based Test Case Selection of Cyber-Physical System Product Lines for Simulation-Based Validation. In The International Systems and Software Product Line Conference (SPLC), pp. 297-306, 2016.
  • A. Arrieta, S. Wang, G. Sagardui and L. Etxeberria. Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms. In The Genetic and Evolutionary Computation Conference (GECCO), Search-Based Software Engineering (SBSE) track, pp. 1053-1060, 2016.
  • D. Pradhan, S. Wang, S. Ali and T. Yue. Search-Based Cost-Effective Test Case Selection for Manual Execution within Time Budget: An Empirical Study. In The Genetic and Evolutionary Computation Conference (GECCO), Search-Based Software Engineering (SBSE) track. pp. 1085-1092, 2016.
  • S. Wang, S. Ali, T. Tue, Ø. Bakkeli, and M. Liaaen. Enhancing Test Case Prioritization in an Industrial Setting with Resource Awareness and Multi-Objective Search. In The 38th International Conference on Software Engineering (ICSE), Software Engineering in Practice (SEIP) track. pp. 182-191, 2016.
  • S. Wang, S. Ali, T. Tue, Y. Li, and M. Liaaen. A Practical Guide to Select Quality Indicators for Assessing Pareto-Based Search Algorithms in Search-Based Software Engineering. In the 38th International Conference on Software Engineering (ICSE). pp. 631-642, 2016.
  • S. Wang, S. Ali, T. Tue, and M. Liaaen. UPMOA: An Improved Search Algorithm to Support User- Preference Multi-Objective Optimization. In The 26th IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 393-404, 2015.
  • S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan and M. Liaaen. Multi-Objective Test Prioritization in Software Product Line Testing: An Industrial Case Study. In: 18th International Software Product Line Conference (SPLC 2014), One of Three Best Paper Nominees, pp. 32-41, 2014.
  • S. Wang, S. Ali, A. Gotlieb. Random-Weighted Search-Based Multi-Objective Optimization Revisited. In: 6th International Symposium on Search-Based Software Engineering (SSBSE 2014), pp. 184-198, 2014.
  • S. Wang, A. Gotlieb, S. Ali, and M. Liaaen. Automated Test Case Selection using Feature Model: An Industrial Case Study, In: ACM/IEEE 16th International Conference on Model Driven Engineering Languages and Systems (MODELS 2013), Best Application Paper Award, pp. 237-253, 2013.
  • S. Wang, S. Ali, T. Yue, and M. Liaaen. Using Feature Model to Support Model-Based Testing of Product Lines: An Industrial Case Study, In: 13th International Conference on Quality Software (QSIC 2013), pp.75-84, 2013.
  • S. Wang, S. Ali, and A. Gotlieb. Minimizing Test Suites in Software Product Lines Using Weight-based Genetic Algorithms, In: ACM Genetic and Evolutionary Computation Conference (GECCO 2013), pp. 1493-1500, 2013.
  • S. Ali, T. Yue, L.C. Briand, and S. Walawege. A product line modeling and configuration methodology to support model-based testing: an industrial case study. In Proceedings of the ACM International Conference Model Driven Engineering Languages and Systems (MODELS). pp. 726-742. 2012.