Here we will discuss the basics of running the test suite. The test suite is written in Java, and commands to the test suite are programmed in cmake. The test suite is able to execute multiple engines in parallel and will provide html reports of the pass/fail status for each of the tests.
The test suite takes in a config file from the source directory. Each config file directs the test suite with instructions.
From a command terminal from the
<pulse/bin> directory run the following command:
# On Windows > run DebugRun # On Linux $ ./run.sh DebugRun
This will run the instructions in the DebugRun.config file located in your
Executing the suite with this config file will do the following :
- Create an engine and run the BasicStandard sceanario where it will write out the scenarios data requests into a
- Compare the generated BasicStandardResults file with the baseline file located at
<pulse/bin>/verification/scenarios/patient/BasicStandardResults.csvThe comparison will compare every value between the two files, if the percent difference between a value is greater than the config files defined
PercentDifferencevariable (usually 2%), an error will be logged. Each column of values has errors associated with it.
- Generate a jpg line plots for each of the data columns. The plots will be located in the
<pulse/bin>/test_results/scenarios/patient/BasicStandardResultsfolder. Columns that matched the baseline file with no errors will generate a jpg outlined in green, while columns that had comparison errors with the baseline file will generate a jpg outlined in red.
<pulse/bin>/test_results/DebugRun.htmlreport is generated to list the pass/fail status of each run in the config file.
|Example of passing and failing plots, from 2 different scenarios.|
Note this execution sequence is only performing verification against the baseline. We will discuss validation tools associated with the test script further below.
All scenarios are specified in the
<pulse/source/data/config>/ScenarioVerification.config file. In general, the best way to run scenarios of your choosing, is to copy the execution line from the Verification.config file into the DebugRun.config file and perform
run DebugRun. You can copy multiple lines into DebugRun and they will execute in parallel. In general we use the DebugRun.config to run scenarios we are working on or who's results we are interested in viewing.
If you want to, and have time, you can run the full verification suite with :
# On Windows > run ScenarioVerification # On Linux $ ./run.sh ScenarioVerification
And all scenarios will run, plots will be populated under the
<pulse/bin>/test_results/scenarios folder, and a ScenarioVerification.html will be generated in the
A Deeper Dive into the Test Suite
Config File Format
The config file format is uses key value pairs to define instructions as follows :
# Run Flags ReportName=Debugging Test Summary ExecuteTests=true PlotResults=true PercentDifference=2.0 Threads=-1 # Executors are the Java classes that can execute a test run type Executor=com.kitware.pulse.testing.ScenarioTestDriver # Macro defines common execution flags in short hand Macro ScenarioTest=ScenarioTestDriver FastPlot Baseline=scenarios/ Computed=./test_results/scenarios # Execution runs are the input and the execution flags, including the executor for the run patient/BasicStandard.json= ScenarioTest
An test run is defined as :
The key (left side of this statement) defines the unit test name or the scenario file name. It is a relative path from the <pulse/bin>/verification directory. The value (right side of this statement) defines execution flags with the Macro
ScenarioTest which defines the
Baseline directory as