FB3
TZI
BISS
Test Generation Benchmarks
This section contains benchmarks of the classification 'test generation benchmarks' and corresponding run-times of the RT-Tester tool whose model-based test generation component RTT-MBT has been used as reference tool for creating the tests.
Overview
Model description
The complete model description is available here.
Mutants description
Test cases description
File testcases specifies all test cases related to model coverage criteria: each model coverage goal is identified by means of a test case tag and a logical formula specifying the constraint for covering the goal. The current baseline of the benchmark model specifies the goals for-
Basic Control State (BCS) coverage
The goals consist in reaching basic control states in (potentially hierarchic) state machines of the model. -
State Machine Transition (TR/HITR) coverage
The goals consist in covering transitions between (higher- or lower-level) control states of state machines. -
MC/DC Coverage
The objective is again to cover state machine transitions, with the refinement that guard conditions a && b should be covered with different cases a && b, a && !b, !a && b, and guard conditions a || b should be covered with test cases a && !b, !a && b, !a && !b, as far as possible (for more complex Boolean expressions the MC/DC criteria are refined in a natural way). -
User defined
The goals are expressed as LTL formula and provided by the user
Test Benchmarks
The benchmarks indicates also if the tests are able to kill the mutants presentedBasic Control State Coverage Tests
-
The following two tests generates 100% BCS coverage. No backtracking or random simulation techniques were applied, only the SMT solver accelerated by abstract interpretation. The following tables refer to the configuration of each test generation process and give short descriptions of what the generated tests do.
TP 001 BCSDescription: Basic control state coverage test procedure Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 0.666s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✘ ✘ ✘ Visualize Test Data
TP 001 BCS NOSBDescription: Basic control state coverage test procedure when no service brake is available Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 0.626s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✘ ✘ ✘ Visualize Test Data State Machine Transition Coverage Tests
-
The following tests generate 100% state machine transition coverage. Two deal with only CSM_ON transitions, whereas the other two deal with all the transition in the model including high-level transition. The NOSB indicates that the test cases consider that the train does not have service brake.
TP 002 OnlyCSM_ONDescription: Transition coverage without high-level transition when service brake is available. Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 1.860s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 002 OnlyCSM_ON NOSBDescription: Transition coverage without high-level transition when no service brake is available Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 3.088s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP 002 Transition NOSBDescription: Transition coverage including high-level transition Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 21.733s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP 002 Transition NOSBDescription: Transition coverage including high-level transition when no service brake is available. Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 8.944s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data State Machine MC/DC Coverage Tests
-
The following tests generate 100% state machine transition coverage. The test procedure with NOSB assumes that no service brake are available.
TP 003 MCDCDescription: MC/DC complete coverage when service brake is available Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 14.773s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✘ ✘ ✘ Visualize Test Data
TP 003 MCDC NOSBDescription: MC/DC complete coverage when no service brake is available Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 16.320s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✘ ✘ ✘ Visualize Test Data Requirement based testing
-
The following tests are requirement based testing. Requirements are link to the model artifacts, transitions, state or constraints (written as LTL formula). The test procedurws are generated from a set of requirements, the tool provide the corresponding tests that ensure that those requirement will be covered. The requirements details may be found here.
TP 004 3.13.10 DMIOutputsDescription: Test procedure covered the requirement related to the driver outputs.
3.13.10.2.1
3.13.10.3.1
3.13.10.3.2Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 13.578s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 004 3.13.10 DMIOutputs NOSBDescription: Test procedure covered the requirement related to the driver outputs when no service brake are available.
3.13.10.2.1
3.13.10.3.1
3.13.10.3.2Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 13.062s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 005 3.13.10 EBvsSBDescription: Test procedure covers the special condition when no service Brake Servicebrake available sbAvailable = 0
3.13.10.2.3
3.13.10.2.4
3.13.10.2.4.r1Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 18.878s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 006 3.13.10 IndicationStatusDescription: Test procedure covers the Indication status value
3.13.10.3.5
3.13.10.3.6Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 3.091s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 006 3.13.10 IndicationStatus NOSBDescription: Test procedure covers the Indication status value when no service brake is available.
3.13.10.3.5
3.13.10.3.6Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 2.985s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✘ ✘ Visualize Test Data
TP 007 3.13.10 RevokeCmdDescription: Test procedure for the revocation of TIcmd and DMICmd
3.13.10.2.5
3.13.10.3.3.r0
3.13.10.3.3.r1
3.13.10.3.4.r1c3
3.13.10.3.4.r1c4
3.13.10.3.4.r1c5Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 4.535s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP 007 3.13.10 RevokeCmd NOSBDescription: Test procedure for the revocation of TIcmd and DMICmd when no service brake is available.
3.13.10.2.5
3.13.10.3.3.r0
3.13.10.3.3.r1
3.13.10.3.4.r1c3
3.13.10.3.4.r1c4
3.13.10.3.4.r1c5Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 3.410s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP 008 3.13.10 TriggerDescription: Test procedure to test the DMICmd and TICmd trigger
3.13.10.3.3.t1
3.13.10.3.3.t2
3.13.10.3.3.t3
3.13.10.3.3.t4
3.13.10.3.3.t5
3.13.10.3.4.r3c1
3.13.10.3.4.r4c1
3.13.10.3.4.r4c3
3.13.10.3.4.r5c1
3.13.10.3.4.r5c3
3.13.10.3.4.r5c4Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 19.049s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP 008 3.13.10 Trigger NOSBDescription: Test procedure to test the DMICmd and TICmd trigger when no service brake is available.
3.13.10.3.3.t1
3.13.10.3.3.t2
3.13.10.3.3.t3
3.13.10.3.3.t4
3.13.10.3.3.t5
3.13.10.3.4.r3c1
3.13.10.3.4.r4c1
3.13.10.3.4.r4c3
3.13.10.3.4.r5c1
3.13.10.3.4.r5c3
3.13.10.3.4.r5c4Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureGeneration Time: 23.636s Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB MemoryMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data The Input Equivalence Class Partioning test suites (written by hands)
-
The following two tests have been written by hand following the definition given in [PH2013]. The fault domain considered here is D(S,m=6, I_2= I).
TP 009 IECPDescription: Complete Test suites for D(S,m=6, I_2= I) alphabet with service brake available Files: Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureMutants: 1 2 3 ✔ ✔ ✔ → Visualize Test Data
TP 009 IECP NOSBDescription: Test suites generated with the IECP method with no service brake available Files: Configuration
Signal Map
Generation Result
Test Covered by the Generated ProcedureMutants: 1 2 3 ✔ ✔ ✔ → Visualize Test Data IECP (Input Equivalence Class Partioning) based testing (auto generated test suite)
-
The following tests were generated by a prototypical implementation of the IECP test generation as described in [PH2013].
This implementation is not able to deal with internal model variables and output variables that have an infinite domain (e.g. floating point variables). Therefore a reduced version of the Ceiling Speed Monitor model has been used. This reduced model does only contain the lower-level state machine CSM_ON modelling the behaviour of the active CSM and the only signals, used in this reduced model, are V_est, V_max as inputs and DMICmd, DMIDisplaySBI, TICmd as outputs. Furthermore it is assumed, that the service brake is available (SBAvailable = 1) and that, due to national regulations, the emergency brakes must only be released after the train has come to a standstill (allowRevokeEB = 0).
TP IECPDescription: The tool calculated 66 input equivalence classes and generated 1630 test procedures.
All test procedures are collected in a single zip file (See below).Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result (zip file of all test procedures)Generation Time: 10 min 21 s Machine: Intel(R) Xeon(R) CPU E5-2667, 2.90GHz
64 GB MemoryThe following tables show two exemplary test procedures that are part of the auto generated IECP test suite. These test procedures can also be found in the zip file linked above, and are just referenced here to give more details.
TP IECP 01040Description: This test procedure starts in control state OVERSPEED, leads to control state NORMAL and finally to control state WARNING. Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result
Test Covered by the Generated ProcedureMutants: 1 2 3 ✔ ✔ ✘ Visualize Test Data
TP IECP 00775Description: This test procedure starts in control state WARNING, leads to control state NORMAL and finally to control state OVERSPEED. Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result
Test Covered by the Generated ProcedureMutants: 1 2 3 ✘ ✘ ✔ Visualize Test Data
[PH2013] Jan Peleska and Wen-ling Huang: Exhaustive Model-Based Equivalence Class Testing. In Yenigün, Hüsnü and Yilmaz, Cemal and Ulrich, Andreas (eds.): Testing Software and Systems, Proceedings of the ICTSS2013. Springer, LNCS 8254, pp.49-64, 2013.
