Updating Baselines

In order to assess any changes to the code base, Pulse provides a Test Suite that will be run to check for any deviations to the engine outputs. These tests execute the engine and output data to a csv file. This generated csv file is then compared to a baseline csv file containing the current accepted results for the same test. Any deviations in the comparison will need to be analyzed to determine if the deviation is an improvement or if a bug was introduced. If the deviation is an improvement, then the baseline csv file will need to be updated along with the code change. This post will detail how to update any test baseline.

The baseline files are located in a Girder Server Each test is associated with a zip file. Each zip file will contain a csv results file. If the zip file is associated with a scenario test, the zip file will also contain the scenario file and a log of the engine execution of that scenario. You must have Write access for this folder in Girder to update the baseline files.

Execute the following steps to update the baseline files. Note, it is assumed you have run the test and have the new results on your computer.

Prepare zip files with the new results to be uploaded to Girder.

You will need to identify the files you wish to rebase by editing some files. The /test/config/Rebase.config is used to reference Test Suite configuration files. If you wish to rebase all the results for tests associated with one or more Test Suite configuration files, simply uncomment the files in this file. If would like to specify individual tests to rebase, you will need to copy the line from the configuration that runs that specific test.

For example, if you wanted to rebase the BrainInjury scenario results. First edit the Rebase.config file to only have the DebugRun.config file uncommented. Then find the config file that contains this test, in this example the test is run via the /test/config/VerificationScenarios.config file. Simply copy the line patient/BrainInjury.pba = ScenarioTest and put it in the /test/config/DebugRun.config file. (Note comment out the BasicStandard line, so you don't rebase that too.)

Once you have described what to rebase in the Rebase.config file, open a command terminal in the pulse build install directory and run the following command :

# On Windows
> run rebase
# On Linux/Max
$ ./run.sh rebase

This will generate all the zip files in a newly created directory under the <pulse_install>/bin/test_results/rebase directory. This directory will be named based on the current date and the current git hash associated with the HEAD of this branch.

Upload Files to Girder

The code to upload data to Girder is written in python and will require that you pip install the following : (Note I highly recommend using virtualenv to separate your python wheels) (You will also need to have python-pip installed)

girder_client

In the same command terminal run the following python script (This assume python is on the path for this command terminal)

$ python rebase.py --api-key xxXXxxXX

Where xxXXxxXX is a girder api key associated with your accout (which has access to write to the verification folder in girder)

Commit and Push Git Changes

As part of the upload process, new SHA512 hashes will be generated and stored in the files in the verification directory. You will need to push these modified files into the repository (or add new ones if you added a new test to the Test Suite)