Universität Bremen  
  Universität Bremen FB3 TZI BISS  
  AG BS > Embedded Systems Testing Benchmarks Site > > Ceiling Speed Monitoring Rev. 1.6 > Deutsch
English
 

Test Generation Benchmarks

 
This section contains benchmarks of the classification 'test generation benchmarks' and corresponding run-times of the RT-Tester tool whose model-based test generation component RTT-MBT has been used as reference tool for creating the tests.

Overview


Model description

The complete model description is available here.

Ceiling speed monioring high
			  level

Ceiling speed monioring
					       CSM_ON state


Mutants description




Mutants



Test cases description

File testcases specifies all test cases related to model coverage criteria: each model coverage goal is identified by means of a test case tag and a logical formula specifying the constraint for covering the goal. The current baseline of the benchmark model specifies the goals for
  • Basic Control State (BCS) coverage
    The goals consist in reaching basic control states in (potentially hierarchic) state machines of the model.

  • State Machine Transition (TR/HITR) coverage
    The goals consist in covering transitions between (higher- or lower-level) control states of state machines.

  • MC/DC Coverage
    The objective is again to cover state machine transitions, with the refinement that guard conditions a && b should be covered with different cases a && b, a && !b, !a && b, and guard conditions a || b should be covered with test cases a && !b, !a && b, !a && !b, as far as possible (for more complex Boolean expressions the MC/DC criteria are refined in a natural way).

  • User defined
    The goals are expressed as LTL formula and provided by the user

For a description of all the configuration file please read this.

Test Benchmarks

The benchmarks indicates also if the tests are able to kill the mutants presented below . The mark ✔ means that the test can kill the mutant, the mark ✘ is depicted otherwise.

Basic Control State Coverage Tests

The following two tests generates 100% BCS coverage. No backtracking or random simulation techniques were applied, only the SMT solver accelerated by abstract interpretation. The following tables refer to the configuration of each test generation process and give short descriptions of what the generated tests do.


TP 001 BCS
Description: Basic control state coverage test procedure
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 0.666s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 001 BCS NOSB
Description: Basic control state coverage test procedure when no service brake is available
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 0.626s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

State Machine Transition Coverage Tests

The following tests generate 100% state machine transition coverage. Two deal with only CSM_ON transitions, whereas the other two deal with all the transition in the model including high-level transition. The NOSB indicates that the test cases consider that the train does not have service brake.


TP 002 OnlyCSM_ON
Description: Transition coverage without high-level transition when service brake is available.
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 1.860s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 002 OnlyCSM_ON NOSB
Description: Transition coverage without high-level transition when no service brake is available
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 3.088s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 002 Transition NOSB
Description: Transition coverage including high-level transition
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 21.733s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 002 Transition NOSB
Description: Transition coverage including high-level transition when no service brake is available.
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 8.944s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

State Machine MC/DC Coverage Tests

The following tests generate 100% state machine transition coverage. The test procedure with NOSB assumes that no service brake are available.


TP 003 MCDC
Description: MC/DC complete coverage when service brake is available
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 14.773s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 003 MCDC NOSB
Description: MC/DC complete coverage when no service brake is available
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 16.320s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

Requirement based testing

The following tests are requirement based testing. Requirements are link to the model artifacts, transitions, state or constraints (written as LTL formula). The test procedurws are generated from a set of requirements, the tool provide the corresponding tests that ensure that those requirement will be covered. The requirements details may be found here.

TP 004 3.13.10 DMIOutputs
Description: Test procedure covered the requirement related to the driver outputs.
3.13.10.2.1
3.13.10.3.1
3.13.10.3.2
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 13.578s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 004 3.13.10 DMIOutputs NOSB
Description: Test procedure covered the requirement related to the driver outputs when no service brake are available.
3.13.10.2.1
3.13.10.3.1
3.13.10.3.2
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 13.062s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 005 3.13.10 EBvsSB
Description: Test procedure covers the special condition when no service Brake Servicebrake available sbAvailable = 0
3.13.10.2.3
3.13.10.2.4
3.13.10.2.4.r1
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 18.878s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 006 3.13.10 IndicationStatus
Description: Test procedure covers the Indication status value
3.13.10.3.5
3.13.10.3.6
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 3.091s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 006 3.13.10 IndicationStatus NOSB
Description: Test procedure covers the Indication status value when no service brake is available.
3.13.10.3.5
3.13.10.3.6
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 2.985s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 007 3.13.10 RevokeCmd
Description: Test procedure for the revocation of TIcmd and DMICmd
3.13.10.2.5
3.13.10.3.3.r0
3.13.10.3.3.r1
3.13.10.3.4.r1c3
3.13.10.3.4.r1c4
3.13.10.3.4.r1c5
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 4.535s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 007 3.13.10 RevokeCmd NOSB
Description: Test procedure for the revocation of TIcmd and DMICmd when no service brake is available.
3.13.10.2.5
3.13.10.3.3.r0
3.13.10.3.3.r1
3.13.10.3.4.r1c3
3.13.10.3.4.r1c4
3.13.10.3.4.r1c5
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 3.410s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 008 3.13.10 Trigger
Description: Test procedure to test the DMICmd and TICmd trigger
3.13.10.3.3.t1
3.13.10.3.3.t2
3.13.10.3.3.t3
3.13.10.3.3.t4
3.13.10.3.3.t5
3.13.10.3.4.r3c1
3.13.10.3.4.r4c1
3.13.10.3.4.r4c3
3.13.10.3.4.r5c1
3.13.10.3.4.r5c3
3.13.10.3.4.r5c4
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 19.049s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

TP 008 3.13.10 Trigger NOSB
Description: Test procedure to test the DMICmd and TICmd trigger when no service brake is available.
3.13.10.3.3.t1
3.13.10.3.3.t2
3.13.10.3.3.t3
3.13.10.3.3.t4
3.13.10.3.3.t5
3.13.10.3.4.r3c1
3.13.10.3.4.r4c1
3.13.10.3.4.r4c3
3.13.10.3.4.r5c1
3.13.10.3.4.r5c3
3.13.10.3.4.r5c4
Files: Test cases to be covered (Goals)
Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Generation Time: 23.636s
Machine: MacBook Pro 2.8 GHz Intel Core i7
16GB Memory
Mutants: 1 2 3
Visualize Test Data

The Input Equivalence Class Partioning test suites (written by hands)

The following two tests have been written by hand following the definition given in [PH2013]. The fault domain considered here is D(S,m=6, I_2= I).


TP 009 IECP
Description: Complete Test suites for D(S,m=6, I_2= I) alphabet with service brake available
Files: Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Mutants: 1 2 3
Visualize Test Data

TP 009 IECP NOSB
Description: Test suites generated with the IECP method with no service brake available
Files: Configuration
Signal Map
Generation Result
Test Covered by the Generated Procedure
Mutants: 1 2 3
Visualize Test Data

IECP (Input Equivalence Class Partioning) based testing (auto generated test suite)

The following tests were generated by a prototypical implementation of the IECP test generation as described in [PH2013].
This implementation is not able to deal with internal model variables and output variables that have an infinite domain (e.g. floating point variables). Therefore a reduced version of the Ceiling Speed Monitor model has been used. This reduced model does only contain the lower-level state machine CSM_ON modelling the behaviour of the active CSM and the only signals, used in this reduced model, are V_est, V_max as inputs and DMICmd, DMIDisplaySBI, TICmd as outputs. Furthermore it is assumed, that the service brake is available (SBAvailable = 1) and that, due to national regulations, the emergency brakes must only be released after the train has come to a standstill (allowRevokeEB = 0).

TP IECP
Description: The tool calculated 66 input equivalence classes and generated 1630 test procedures.
All test procedures are collected in a single zip file (See below).
Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result (zip file of all test procedures)
Generation Time: 10 min 21 s
Machine: Intel(R) Xeon(R) CPU E5-2667, 2.90GHz
64 GB Memory

The following tables show two exemplary test procedures that are part of the auto generated IECP test suite. These test procedures can also be found in the zip file linked above, and are just referenced here to give more details.

TP IECP 01040
Description: This test procedure starts in control state OVERSPEED, leads to control state NORMAL and finally to control state WARNING.
Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result
Test Covered by the Generated Procedure
Mutants: 1 2 3
Visualize Test Data

TP IECP 00775
Description: This test procedure starts in control state WARNING, leads to control state NORMAL and finally to control state OVERSPEED.
Files: Reduced CSM Model used for IECP test case generation (XMI format)
Reduced CSM Model used for IECP test case generation (HTML export)
Reduced CSM Model used for IECP test case generation (EAP format)
Signal Map
Generation Result
Test Covered by the Generated Procedure
Mutants: 1 2 3
Visualize Test Data


[PH2013] Jan Peleska and Wen-ling Huang: Exhaustive Model-Based Equivalence Class Testing. In Yenigün, Hüsnü and Yilmaz, Cemal and Ulrich, Andreas (eds.): Testing Software and Systems, Proceedings of the ICTSS2013. Springer, LNCS 8254, pp.49-64, 2013.
     
Author: uschulze     AG BS 
Last updated: October 16, 2015   Impressum