Here we will discuss the basics of running the test suite. The test suite is written in Java, and commands to the test suite are programmed in cmake. The test suite is able to execute multiple engines in parallel and will provide html reports of the pass/fail status for each of the tests.

The test suite takes in a config file from the source directory. Each config file directs the test suite with instructions.

Quick Start

From a command terminal from the <pulse/bin> directory run the following command:

# On Windows
> run DebugRun
# On Linux
$ ./run.sh DebugRun

This will run the instructions in the DebugRun.config file located in your <pulse/source>/data/config directory.

Executing the suite with this config file will do the following :

1. Create an engine and run the BasicStandard sceanario where it will write out the scenarios data requests into a <pulse/bin>/test_results/scenarios/patient/BasicStandardResults.csv file.
2. Compare the generated BasicStandardResults file with the baseline file located at <pulse/bin>/verification/scenarios/patient/BasicStandardResults.csv The comparison will compare every value between the two files, if the percent difference between a value is greater than the config files defined PercentDifference variable (usually 2%), an error will be logged. Each column of values has errors associated with it.
3. Generate a jpg line plots for each of the data columns. The plots will be located in the <pulse/bin>/test_results/scenarios/patient/BasicStandardResults folder. Columns that matched the baseline file with no errors will generate a jpg outlined in green, while columns that had comparison errors with the baseline file will generate a jpg outlined in red.
4. A <pulse/bin>/test_results/DebugRun.html report is generated to list the pass/fail status of each run in the config file.

Example of passing and failing plots, from 2 different scenarios.

Note this execution sequence is only performing verification against the baseline. We will discuss validation tools associated with the test script further below.

All scenarios are specified in the <pulse/source/data/config>/ScenarioVerification.config file. In general, the best way to run scenarios of your choosing, is to copy the execution line from the Verification.config file into the DebugRun.config file and perform run DebugRun. You can copy multiple lines into DebugRun and they will execute in parallel. In general we use the DebugRun.config to run scenarios we are working on or who's results we are interested in viewing.

If you want to, and have time, you can run the full verification suite with :

# On Windows
> run ScenarioVerification
# On Linux
$ ./run.sh ScenarioVerification

And all scenarios will run, plots will be populated under the <pulse/bin>/test_results/scenarios folder, and a ScenarioVerification.html will be generated in the <pulse/bin>/test_results folder.

A Deeper Dive into the Test Suite

Config File Format

The config file format is uses key value pairs to define instructions as follows :

# Run Flags
ReportName=Debugging Test Summary
ExecuteTests=true
PlotResults=true
PercentDifference=2.0
Threads=-1
# Executors are the Java classes that can execute a test run type
Executor=com.kitware.pulse.testing.ScenarioTestDriver
# Macro defines common execution flags in short hand
Macro ScenarioTest=ScenarioTestDriver FastPlot Baseline=scenarios/ Computed=./test_results/scenarios
# Execution runs are the input and the execution flags, including the executor for the run
patient/BasicStandard.json= ScenarioTest

An test run is defined as :

patient/BasicStandard.json= ScenarioTest

The key (left side of this statement) defines the unit test name or the scenario file name. It is a relative path from the <pulse/bin>/verification directory. The value (right side of this statement) defines execution flags with the Macro ScenarioTest which defines the Baseline directory as scenarios