|
|
Here we will discuss the basics of running the test suite. The test suite is written in Java, and commands to the test suite are programmed in cmake. The test suite is able to execute multiple engines in parallel and will provide html reports of the pass/fail status for each of the tests.
|
|
|
|
|
|
The test suite takes in a [config file](https://gitlab.kitware.com/physiology/engine/tree/master/test/config) from the source directory. Each config file directs the test suite with instructions.
|
|
|
|
|
|
Quick Start
|
|
|
-----------
|
|
|
From a command terminal from the `<pulse/bin>` directory run the following command:
|
|
|
```bash
|
|
|
# On Windows
|
|
|
> run DebugRun
|
|
|
# On Linux
|
|
|
$ ./run.sh DebugRun
|
|
|
```
|
|
|
This will run the instructions in the [DebugRun.config](https://gitlab.kitware.com/physiology/engine/blob/master/test/config/DebugRun.config) file located in your `<pulse/source>/test/config` directory.
|
|
|
|
|
|
Executing the suite with this config file will do the following :
|
|
|
1. Create an engine and run the BasicStandard sceanario where it will write out the scenarios data requests into a `<pulse/bin>/test_results/scenarios/patient/BasicStandardResults.txt` file.
|
|
|
2. Compare the generated BasicStandardResults file with the baseline file located at `<pulse/bin>/verification/scenarios/patient/BasicStandardResults.txt` The comparison will compare every value between the two files, if the percent difference between a value is greater than the config files defined `PercentDifference` variable (usually 2%), an error will be logged. Each column of values has errors associated with it.
|
|
|
3. Generate a jpg line plots for each of the data columns. The plots will be located in the `<pulse/bin>/test_results/scenarios/patient/BasicStandardResults` folder. Columns that matched the baseline file with no errors will generate a jpg outlined in green, while columns that had comparison errors with the baseline file will generate a jpg outlined in red.
|
|
|
4. A `<pulse/bin>/test_results/DebugRun.html` report is generated to list the pass/fail status of each run in the config file.
|
|
|
|
|
|
Note this execution sequence is only performing verification against the baseline. We will discuss validation tools associated with the test script further below.
|
|
|
|
|
|
All scenarios are specified in the `<pulse/source/test/config>/`[VerificationScenarios.config](https://gitlab.kitware.com/physiology/engine/blob/master/test/config/VerificationScenarios.config) file. In general, the best way to run scenarios of your choosing, is to copy the execution line from the Verification.config file into the DebugRun.config file and perform `run DebugRun`. You can copy multiple lines into DebugRun and they will execute in parallel. In general we use the DebugRun.config to run scenarios we are working on or who's results we are interested in viewing.
|
|
|
|
|
|
If you want to, and have time, you can run the full verification suite with :
|
|
|
```bash
|
|
|
# On Windows
|
|
|
> run VerificationScenarios
|
|
|
# On Linux
|
|
|
$ ./run.sh VerificationScenarios
|
|
|
```
|
|
|
And all scenarios will run, plots will be populated under the `<pulse/bin>/test_results/scenarios` folder, and a VerificationScenarios.html will be generated in the `<pulse/bin>/test_results folder`.
|
|
|
|
|
|
A Deeper Dive into the Test Suite
|
|
|
=================================
|
|
|
|
|
|
Config File Format
|
|
|
------------------
|
|
|
The config file format is uses key value pairs to define instructions as follows :
|
|
|
|
|
|
```
|
|
|
# Run Flags
|
|
|
ReportName=Debugging Test Summary
|
|
|
ExecuteTests=true
|
|
|
PlotResults=true
|
|
|
PercentDifference=2.0
|
|
|
Threads=-1
|
|
|
# Executors are the Java classes that can execute a test run type
|
|
|
Executor=mil.tatrc.physiology.datamodel.testing.CDMUnitTestDriver
|
|
|
# Macro defines common execution flags in short hand
|
|
|
Macro ScenarioTest=ScenarioTestDriver FastPlot Baseline=scenarios/ Computed=./test_results/scenarios
|
|
|
# Execution runs are the input and the execution flags, including the executor for the run
|
|
|
patient/BasicStandard.pba = ScenarioTest
|
|
|
```
|
|
|
|
|
|
An test run is defined as :
|
|
|
```
|
|
|
patient/BasicStandard.pba = ScenarioTest
|
|
|
```
|
|
|
The key (left side of this statement) defines the unit test name or the scenario file name. It is a relative path from the <pulse/bin>/verification directory. The value (right side of this statement) defines execution flags with the Macro `ScenarioTest` which defines the `Baseline` directory as `scenarios`
|
|
|
|
|
|
More to come...
|
|
|
|