Enhance validation process
Summary
The command validate-system doesn't currently include the validation of the self-reported outputs
Steps to reproduce
Well just run validate-system and see by yourself
What is the expected correct behavior?
The validate-system command should validate three things:
- That the
container_outputs
directory contains at least one system output, - That each reported system output is different from the baseline's,
- The directory name starts with a dataset name
These three checks should be performed all the time, and only cause a failure if a --strict
flag is provided.
Possible fixes
Look at this, it contains the function to compare two system outputs: https://gitlab.kitware.com/alexandreB/indus-config/blob/master/step_actev_validate_nist_execution.sh
Move the baseline system output from container_outputs to bin/...
Edited by Maxime Hubert