... | ... | @@ -125,7 +125,7 @@ $./run.sh LongVerificationScenarios |
|
|
$./run.sh ScenarioVerification
|
|
|
```
|
|
|
|
|
|
### Verification Reporting
|
|
|
### CSV Verification Reporting
|
|
|
|
|
|
For any test that generates a CSV file. There will be a baseline CSV file under the `<pulse/bin>/verification` directory. (If you do not have these, execute `run updateBaselines` from a terminal in the ~<pulse/bin>` directory.) Once the suite executes a test with a CSV file, a post processor will perform the following tasks.
|
|
|
|
... | ... | @@ -151,8 +151,58 @@ Once we are happy with how our model updates are performing with these scenarios |
|
|
Validation
|
|
|
----------
|
|
|
|
|
|
New feature and improvement to Pulse will inevitably create scenario test failures. Some failures will be minimal and plots will show failures where the generated data is still looks the same as the expected data, other failures will show very obvious changes in data trends. For example you may be improving the model to increase the etCO2 during a specific insult. The desired increase will result in a scenario failure. This update may also cause a failures in other scenarios where the updated model is not active. These failures may or may not be acceptable. To know how a scenario a failing scenario is still valid, you must look up its validation data and manually confirm that the deviating data still peforms as expected. Scenario validation tables are kept in spreadsheets that can be found in `<pulse/source>/data/human/adult/validation/Scenarios`. You must examine that each failing scenario still maintains valid trends according to these tables.
|
|
|
|
|
|
The provided test suite is able to automatically check and report the validity of the homeostatic basline values of all systems and patients. To perform this validation, you can run the following commands
|
|
|
|
|
|
```bash
|
|
|
$./run.sh SystemValidation
|
|
|
$./run.sh PatientValidation
|
|
|
```
|
|
|
|
|
|
### System Validaton
|
|
|
|
|
|
The test suite will run the SystemVerification.config file and generate a CSV file for each system verification scenario. Next, the CSV file will be read along with the SystemValidationData.xlsx files. min/max/means will be calculated from the CSV file, and compared to the expected values in the xlsx file.
|
|
|
A set of HTML tables will be made showing how all system data compares to the validation values. This table is color coded where anything less that 10% error is acceptable and green, anything between 10% and 30% is yellow and considered weak acceptance, and anything over 30% is red and show where we do not meet validation.
|
|
|
|
|
|
<img src="uploads/9265210b04e02a39edb75a45f5a5ced1/image.png" width="50%"/>
|
|
|
|
|
|
We manually compare these new validation tables with the previous validation tables to see what parameters have shifted to decide if the changes are accepted or not.
|
|
|
|
|
|
### Patient Validation
|
|
|
|
|
|
The test suite will run the PatientVerification.config file and generate a CSV file for each patient. Next, the CSV file will be read along with the PatientValidationData.xlsx files. min/max/means will be calculated from the CSV file, and compared to the expected values in the xlsx file.
|
|
|
A set of HTML tables will be made showing how all patient data compares to the validation values. This table is color coded where anything less that 10% error is acceptable and green, anything between 10% and 30% is yellow and considered weak acceptance, and anything over 30% is red and show where we do not meet validation.
|
|
|
|
|
|
<img src="uploads/f42146defe74ffd8fde9dfcb19911481/image.png" width="50%"/>
|
|
|
|
|
|
We manually compare these new validation tables with the previous validation tables to see what parameters have shifted to decide if the changes are accepted or not.
|
|
|
|
|
|
Full Testing
|
|
|
------------
|
|
|
|
|
|
To run ALL testing (CDM and Pulse Unit tests, all verification scenarios, system and patient validation) you can run the following command
|
|
|
|
|
|
```bash
|
|
|
$run tests
|
|
|
```
|
|
|
This will take up most of your processor and take several hours, but at the end, you will have a single html report with all test results.
|
|
|
|
|
|
<table border="2" align="center">
|
|
|
<tr>
|
|
|
<th colspan="2">Test Resuls directory and the full final test report</th>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td align="left" valign="center">
|
|
|
<img src="uploads/a6e5f92aeff5abd362d92b59473ec147/image.png" width="300" height="200">
|
|
|
</td>
|
|
|
<td align="right" valign="center">
|
|
|
<img src="uploads/c09288789593e7e7066379b95380aea1/image.png" width="300" height="200">
|
|
|
</td>
|
|
|
</tr>
|
|
|
</table>
|
|
|
|
|
|
Rebasing
|
|
|
--------
|
|
|
|
|
|
Once you have made changes to Pulse and have fully investigated and any failing test cases are still producing valid results, you will need to rebase the verification baseline csv files](https://gitlab.kitware.com/physiology/engine/wikis/Updating%20Baselines). This needs to be done via a Pulse maintaining member. |
|
|
\ No newline at end of file |
|
|
Once you have made changes to Pulse and have fully investigated and any failing test cases are still producing valid results, you will need to [rebase the verification baseline csv files](https://gitlab.kitware.com/physiology/engine/wikis/Updating%20Baselines). This needs to be done via a Pulse maintaining member. |
|
|
\ No newline at end of file |