Project : vertecs
Section: Scientific Foundations
Automatic Test Generation
Conformance testing consists in checking whether an implementation under test (abbreviated as IUT) behaves correctly with respect to its specification. In the line of model-based testing, we use formal specifications and their underlying models to unambiguously define conformance testing and test case generation. One difficult problem is to generate adequate test cases (the selection problem) that correctly identify faults (the oracle problem). We use test purposes for selection, which allow to generate tests targeted to specific behaviours. For solving the oracle problem we adapt a well established theory of conformanced testing , which formally defines conformance as a relation between formal models of specifications and implementations. This conformance relation, called ioco compares suspension traces of the specification and implementation. Suspension traces are sequence of input, output or quiescence (absence of action), thus abstract away internal behaviors that cannot be observed by testers. Roughly speaking, an implementation is conformant if after a suspension trace of the specification, the implementation can only show outputs and quiescences of specfication.
We use IOLTS (or IOSTS) as formal models for specifications, implementations, test purposes, and test cases. Most often, specifications are not directly given in such low-level models, but are written in higher-level specification languages (e.g. SDL, UML, Lotos). The tools associated with these languages often contain a simulation API that implements their semantics under the form of IOLTS. On the other hand, the IOSTS model is expressive enough to allow a direct representation of most constructs of the higher-level languages. Test purposes are specified directly as IOLTS (or IOSTS). They are associated with marked states, giving them the status of automata or observers. When IOLTS are considered, test purposes are accepting sequences of actions whose projection on visible actions are selected for test generation. When IOSTS are considered, test purposes are extended automata that observe behaviors, i.e. sequences of actions and vectors of variables of the specifications, whose projection is selected for test generation.
A test case produces a verdict when executed on an implementation. These are formalized in IOLTS (or IOSTS) by special states (or locations). A Fail verdict means that the IUT is rejected, a Pass verdict means that the IUT exhibited a correct behavior and the test purpose has been satisfied, while an Inconclusive verdict is given to a correct behavior that is not accepted by the test purpose. Based on these models, an interaction semantics, and the conformance relation, one can then define correctness properties of test cases and test suites (sets of test cases). Typical properties are soundness (no conformant implementation may be rejected) and exhaustiveness (every non conformant implementation may be rejected).
We have developed test generation algorithms. These algorithms are based on the construction of a product between the specification and test purpose, the computation of its visible behaviors involving the identification of quiescence and determinization, and the selection of accepted behaviors. Selection can be seen as a model-checking problem where one wants to identify states (and transition between them) that are reachable from the initial state and co-reachable from the accepting states. We have proved that these algorithms ensure that the (infinite) set of all possibly generated test cases is sound and exhaustive.
Apart from these theoretical results, our algorithms are designed to be as efficient as possible in order to be able to scale up to real applications. Roughly speaking, test generation consists in computing the intersection of the observable behavior of the specification (traces and quiescence) and the language accepted by the test purpose. The computation of observable behaviors involves determinization, while language intersection is based on computing the set of states that are reachable from initial states and co-reachable states from accepting states.
Our first test generation algorithms are based on enumerative techniques, optimized to fight the state-space explosion problem. We have developed on-the-fly algorithms, which consist in performing a lazy exploration of the set of states that are reachable in both the specification and the test purpose. The resulting test case is an IOLTS whose traces describe interactions with an implementation under test. This technique is now mature and implemented in the TGV tool, which we often use in industrial collaborations. We are continuously improving the technique. However, what characterizes this enumerative technique is that values of variables and communication parameters are instantiated at test generation time.
More recently, we have explored symbolic test generation techniques. This is a promising technique whose main objective is to avoid the state space explosion problem induced by the enumeration of values of variables and communication parameters. The idea consists in computing a test case under the form of an IOSTS, i.e., a reactive program in which the operations on data is kept in a symbolic form. However, most of the operations involved in test generation (determinization, reachability, and coreachability) become undecidable. For determinization we employ heuristics that allow us to solve the so-called bounded observable non-determinism (i.e., the result of an internal choice can be detected after finitely many observable actions). For the remaining operations, reachable and co-reachable sets of states are approximated using abstract interpretation techniques. These techniques are implemented in the STG tool.