RTCM: A Restricted Natural Language Based Test Case Generation Approach


Overview

RTCM is a deliberate extension of Zen-RUCM for automated generation of test cases. Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we propose our solution, a TCS language, named as Restricted est Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Furthemore, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM.


RTCM Methodology

  • Domain Model

    domainModelPig

    RTCM is inspired by Zen-RUCM in terms of specifying scenarios. As shown in the domain model, the essential idea of ScenarioSpecification is that a flow of events is composed of a sequence of steps (Sentences) and a PostCondition. Flows of events are further classified into BasicFlow andAlternativeFlow, which is further classified into four types: SpecificAlternative, BoundedAlternative, GlobalAlternative andOracleVerificationFlow. OracleVerificationFlow is newly proposed for RTCM and the other three ones are from RUCM. TestItem is a concept borrowed from the ISO/IEC/IEEE 29119 standard[1], where TestItem (an alternate to commonly used term SUT) is defined as “work product that is an object of testing”. The test item is composed of a set of TestSetup and TestCaseSpecification, both of which should be specified using the ScenarioSpecification template. TestSetup can be shared across TCSs. We define TestSetup as a set of steps to get the test item ready for executing flows of events defined in a TCS. TestCaseSpecification, as defined in [1], is “documentation of a set of one or more test cases”.

    [1] ISO/IEC/IEEE, 2013. Software and systems engineering-Software testing-Part 1: Concepts and definitions. ISO/IEC/IEEE 29119-1:2013(E), Switzerland.

  • RTCM Template

    templatePig

  • New Keywords
    • INCLUDE TEST CASE SPECIFICATION

      Keyword 'INCLUDE TEST CASE SPECIFICATION' is defined to specify that a TCS can refer to another one to facilitate the reuse of specification fragments. This keyword, from the tooling perspective, corresponds to the Include semantics of use case diagrams and can therefore be formalized in TCMeta to facilitate automated transformation to test cases.

    • INVOKES API

      Keyword 'INVOKES API' is particularly defined for RTCM to facilitate the generation of automatically executable test cases.

    • VERIFIES THAT

      We define keyword 'VERIFIES THAT' to provide a way for test engineers to manually verify a test sequence step and therefore subjects of sentences with this keyword must be Tester. This keyword will be used during the generation of test cases as an indication of manual steps that have to be performed.

  • Modelling Guidelines
      RTCM can be used at least in the following two ways to support various testing activities:
    • Specify high-level TCSs as documentation to enable effective communication among stakeholders. Some of the features of RTCM (e.g., InvokeAPI) might not be useful for this purpose, which simplifies the application of RTCM in this context. Moreover, the NL aspect of RTCM eases its adoption in practice.
    • Specify low-level TCSs to enable automated generation of manually executable test cases. In this context, it is often not needed to use keyword INVOKE API. Keyword VERIFY THAT should be frequently used to specify what a tester should manually check at a specific point of time. Such low-level TCSs can of course be used for the communication purpose. Considering that RTCM is structured and restricted, it is inherently more precise than free text. Notice that in certain context (e.g., the environment stress testing in the avionics domain), test cases can only be manually executed by test engineers and a precise documentation of testing steps in an easy-to-understand manner is therefore very import RTCM can be considered as a modeling methodology to enable MBT.

Tool Support (aTOUCAN4Test)

  • An overview of aToucan4Test

    templatePig

    TCSs are specified in the RTCM editor implemented in our modeling framework called Lightweight Modeling Framework. Based on the LMF framework, we implemented RTCMEditor and it was used to conduct the case studies. Keywords can be automatically highlighted and auto fill of API, status and configuration of the test item is also supported. Moreover, we embedded domain specific APIs, state variables, and configuration parameters relevant to our current industrial partner to ease specifying TCSs. However, this is a specific feature that we enabled in this industrial case study and can be disabled for other industrial applications and thus doesn’t affect the generalization of the approach to other contexts. Notice that this feature can also be disabled if the RTCM editor is used to specify TCSs not targeting for the automated generation of automatically executable test cases.

  • Platform Independent Test Generator (PITG)

    The main functionality of this transformation is to instantiate TCSs written in RTCMEditor into instances of TCMeta such that transformation to executable test cases can be facilitated in a specific test scripting language. The partial instantiation of to facilitate cost-effective manual testing.

  • Platform Specific Test Generator (PSTG)

    As shown in the above overview of aToucan4Test, PSTG takes input InstantiatedTCS and outputs executable test cases, based on various coverage criteria on RTCM specifications (e.g., structural coverage and data coverage). The coverage criteria implemented in PSTG inspired from traditional testing coverage criteria [12; 23]. We defined two types of coverage criteria: Structural Coverage and Data Coverage. Structural coverage criteria focus on covering certain structural features of RTCM specifications. We implemented these criteria: 1) All Branch Coverage: This coverage criterion generates a set of test cases that covers all branches of RTCM specifications at least once;

    • INCLUDE TEST CASE SPECIFICATION

      Keyword 'INCLUDE TEST CASE SPECIFICATION' is defined to specify that a TCS can refer to another one to facilitate the reuse of specification fragments. This keyword, from the tooling perspective, corresponds to the Include semantics of use case diagrams and can therefore be formalized in TCMeta to facilitate automated transformation to test cases.

    • INVOKES API

      Keyword 'INVOKES API' is particularly defined for RTCM to facilitate the generation of automatically executable test cases.

    • VERIFIES THAT

      We define keyword 'VERIFIES THAT' to provide a way for test engineers to manually verify a test sequence step and therefore subjects of sentences with this keyword must be Tester. This keyword will be used during the generation of test cases as an indication of manual steps that have to be performed.


Publications

  • Tao Yue, Shaukat Ali, and Man Zhang. Applying A Restricted Natural Language Based Test Case Generation Approach in An Industrial Context, In International Symposium on Software Testing and Analysis (ISSTA), 2015.
  • Man Zhang, Tao Yue, Shaukat Ali, Huihui Zhang, and Ji Wu. A Systematic Approach to Automatically Derive Test Cases from Use Cases Specified in Restricted Natural Languages, In: 8th System Analysis and Modelling Conference (SAM'14).