|
|
Here we will discuss the basics of running the test suite. The test suite is written in Java, and commands to the test suite are programmed in cmake. The test suite is able to execute multiple engines in parallel and will provide html reports of the pass/fail status for each of the tests.
|
|
|
Here we will discuss the basics of running the test suite. A script is provided to run all of our tests, utilities, and tools. We provide a simple `run` batch/shell script wrapper around a CMake file to ensure cross platform compatibility. **CMake should be available on your path to use this tool.** To call this script, your terminal must be in the `<pulse/build>/install/bin`, which we will refer to as `<pulse/bin>` and calling format to be run is as follows :
|
|
|
|
|
|
The test suite takes in a [config file](https://gitlab.kitware.com/physiology/engine/tree/master/data/config) from the source directory. Each config file directs the test suite with instructions.
|
|
|
|
|
|
Quick Start
|
|
|
-----------
|
|
|
From a command terminal from the `<pulse/install/bin>` directory run the following command:
|
|
|
```bash
|
|
|
# On Windows
|
|
|
> run DebugRun
|
|
|
# On Linux
|
|
|
$ ./run.sh DebugRun
|
|
|
```
|
|
|
This will run the instructions in the [DebugRun.config](https://gitlab.kitware.com/physiology/engine/blob/master/data/config/DebugRun.config) file located in your `<pulse/source>/data/config` directory.
|
|
|
|
|
|
Executing the suite with this config file will do the following :
|
|
|
1. Create an engine and run the BasicStandard sceanario where it will write out the scenarios data requests into a `<pulse/bin>/test_results/scenarios/patient/BasicStandardResults.csv` file.
|
|
|
2. Compare the generated BasicStandardResults file with the baseline file located at `<pulse/bin>/verification/scenarios/patient/BasicStandardResults.csv` The comparison will compare every value between the two files, if the percent difference between a value is greater than the config files defined `PercentDifference` variable (usually 2%), an error will be logged. Each column of values has errors associated with it.
|
|
|
3. Generate a jpg line plots for each of the data columns. The plots will be located in the `<pulse/bin>/test_results/scenarios/patient/BasicStandardResults` folder. Columns that matched the baseline file with no errors will generate a jpg outlined in green, while columns that had comparison errors with the baseline file will generate a jpg outlined in red.
|
|
|
4. A `<pulse/bin>/test_results/DebugRun.html` report is generated to list the pass/fail status of each run in the config file.
|
|
|
|
|
|
<table border="2" align="center">
|
|
|
<tr>
|
|
|
<th colspan="2">Example of passing and failing plots, from 2 different scenarios.</th>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td align="left" valign="center">
|
|
|
<img src="/uploads/9634b3e6e99e44320987624b4c414d1c/MeanArterialPressurevsTime.jpg" width="300" height="200">
|
|
|
</td>
|
|
|
<td align="right" valign="center">
|
|
|
<img src="/uploads/c473668aa75aee60a76f1230a458d06b/MeanArterialPressurevsTime.jpg" width="300" height="200">
|
|
|
</td>
|
|
|
</tr>
|
|
|
</table>
|
|
|
|
|
|
Note this execution sequence is only performing verification against the baseline. We will discuss validation tools associated with the test script further below.
|
|
|
|
|
|
All scenarios are specified in the `<pulse/source/data/config>/`[ScenarioVerification.config](https://gitlab.kitware.com/physiology/engine/blob/master/data/config/ScenarioVerification.config) file. In general, the best way to run scenarios of your choosing, is to copy the execution line from the Verification.config file into the DebugRun.config file and perform `run DebugRun`. You can copy multiple lines into DebugRun and they will execute in parallel. In general we use the DebugRun.config to run scenarios we are working on or who's results we are interested in viewing.
|
|
|
|
|
|
If you want to, and have time, you can run the full verification suite with :
|
|
|
```bash
|
|
|
# On Windows
|
|
|
> run ScenarioVerification
|
|
|
# On Linux
|
|
|
$ ./run.sh ScenarioVerification
|
|
|
> run [command]
|
|
|
# On Linux/Mac
|
|
|
# Make sure to chmod to run this script on your machine
|
|
|
$ chmod +x run.sh
|
|
|
# Once done, you may execute any command
|
|
|
$ ./run.sh [command]
|
|
|
```
|
|
|
And all scenarios will run, plots will be populated under the `<pulse/bin>/test_results/scenarios` folder, and a ScenarioVerification.html will be generated in the `<pulse/bin>/test_results folder`.
|
|
|
|
|
|
A Deeper Dive into the Test Suite
|
|
|
=================================
|
|
|
|
|
|
Config File Format
|
|
|
------------------
|
|
|
The config file format is uses key value pairs to define instructions as follows :
|
|
|
The followings **[command]**s are used to generate data:
|
|
|
|
|
|
```
|
|
|
# Run Flags
|
|
|
ReportName=Debugging Test Summary
|
|
|
ExecuteTests=true
|
|
|
PlotResults=true
|
|
|
PercentDifference=2.0
|
|
|
Threads=-1
|
|
|
# Executors are the Java classes that can execute a test run type
|
|
|
Executor=com.kitware.pulse.testing.ScenarioTestDriver
|
|
|
# Macro defines common execution flags in short hand
|
|
|
Macro ScenarioTest=ScenarioTestDriver FastPlot Baseline=scenarios/ Computed=./test_results/scenarios
|
|
|
# Execution runs are the input and the execution flags, including the executor for the run
|
|
|
patient/BasicStandard.json= ScenarioTest
|
|
|
```
|
|
|
- **genData**- Generate data files needed for stabilizing a patient. This reads the `<pulse/source>/data/Data.xlxs` file (containing Patients, Substances, etc.) and generates files into several directories in the `<pulse/bin>` directory.
|
|
|
|
|
|
An test run is defined as :
|
|
|
```
|
|
|
patient/BasicStandard.json= ScenarioTest
|
|
|
```
|
|
|
The key (left side of this statement) defines the unit test name or the scenario file name. It is a relative path from the <pulse/bin>/verification directory. The value (right side of this statement) defines execution flags with the Macro `ScenarioTest` which defines the `Baseline` directory as `scenarios` |
|
|
- **genStates**- Generate a state file for every patient in the `<pulse/bin/patients>` directory. A new state file is saved at the point where the patient completes stabilization (Simulation time 0s) to the `<pulse/bin>/states` directory
|
|
|
|
|
|
- **updateBaselines**- Pull log and CSV files for all verifiation tests. These CSV files are the expected results of all tests. These files can change frequently. When you are developing on Pulse and pull from the repository, you should also run this command to ensure you are comparing your generated results with the latest baselines. Specifically if anything changes under this [folder](https://gitlab.kitware.com/physiology/engine/-/tree/master/data/human/adult/baselines)
|
|
|
|
|
|
- **protoc**- This will recompile all google proto files. When you are developing on Pulse and modify a proto file, you can run this command to ensure your changes work. If you pull new proto file updates from the repository, the CMake ZERO_CHECK step will detect new proto files and automatically run the protoc compiler.
|
|
|
|
|
|
- **jar**- This will rebuild the Pulse.jar file
|
|
|
|
|
|
- **setup.py**- This copies all needed python files to `<pulse/install/python>` so you can just add that directory to your PYTHONPATH if desired.
|
|
|
|
|
|
- **doxygen**- Generate the html we use for our website. You will need to have your [system properly configured for this to work](Generating-Documentation)
|
|
|
|
|
|
The followings **[command]**s are used to execute several utility tools:
|
|
|
|
|
|
- **rebase**- Used to generate new baselines for tests. [More infor can be found here](Updating-Baselines)
|
|
|
|
|
|
- **fullreport**- This generates one html report file from all html reports files.
|
|
|
|
|
|
- **plotter**- Takes in a CSV file and generates a png line plot for each header in the CSV file where the header is the Y-axis vs Time is the X-axis
|
|
|
|
|
|
- **compare**- Takes in two CSV files and generates a png line plot for each header in the first CSV file. The header is the Y-axis vs Time is the X-axis. Each png line plot will have 2 lines, 1 from each CSV (if the CSV 2 does not have the header from CSV 1, the plot will only have 1 line)
|
|
|
|
|
|
- **plot**- This will process a plot json file and create all plots it specifies.
|
|
|
|
|
|
The followings **[command]**s are used to execute our test suite:
|
|
|
|
|
|
- **SystemValidation**- Will use `<pulse/source>/test/config/ValidationSystems.config` which will create a CSV file per system. Next, each CSV file will be read in by the test suite, and it will compute and compare the min/max/mean of values and check how close they are to valid values provided in our sytem validation spreadsheet (`<pulse/source/data/human/adult/validation/SystemValidationData.xlsx`) and create system validation tables in the `<pulse/bin>/test_results` directory
|
|
|
|
|
|
- **PatientValidation**- Will use `<pulse/source>/test/config/ValidationPatients.config` which will create a CSV file per patient. Next, each CSV file will be read in by the test suite, and it will compute and compare the min/max/mean of values and check how close they are to valid values provided in our patient validation spreadsheet (`<pulse/source/data/human/adult/validation/PatientValidationData.xlsx`) and create patioent validation tables in the `<pulse/bin>/test_results` directory
|
|
|
|
|
|
- **The names of any `*.config` file** in the `<pulse/source>/test/config/` directory (don't include the extension)
|
|
|
- For Example : DebugRun, CDMUnitTests, ValidationSystems, VerificationScenarios, etc.
|
|
|
- With exception of the Characterize.config, this is a legacy testing configuration that will not work
|
|
|
|
|
|
- **Tests**- This will run all tests available in Pulse, this may take several hours depending on your system. This will also use all cores on your system.
|
|
|
|
|
|
More info on running our test suite can be found [here](Verification%20and%20Validation) |