actev validate-execution command: add a flag to validate/score an output
What is the current behavior?
The actev validate-execution
command is currently able to
validate and score an output against a reference as follows:
actev validate-execution
-o <path to output result>
-r <path to reference file>
-a <path to activity>
-f <path to file>
-R <path to scoring result>
What is the desired behavior?
It would be great to add a flag --score
to the actev validate-execution
.
The new default behavior would only validate the output without scoring it:
actev validate-execution
-o <path to output result>
-a <path to activity>
-f <path to file>
The command using the --score
flag would validate and score the output:
actev validate-execution
-o <path to output result>
-r <path to reference file>
-a <path to activity>
-f <path to file>
-R <path to scoring result>
--score
Changes
- In
/diva_evaluation_cli/bin/commands/actev_validate_execution.py
:- Add an argument
--score
in thevalidate-execution
parser in/bin/commands/actev-validate-execution.py
- Remove
-r --reference
and -R --results` arguments from the required argument group: they are not mandatory anymore
- Add an argument
- In
/diva_evaluation_cli/bin/private_src/entry_points/actev_validate_execution.py
:- Add a new parameter in the function signature:
score
- Integrate the argument to the
script
variable that is passed to theimplementation
part.
- Add a new parameter in the function signature:
- In
/diva_evaluation_cli/bin/private_src/implementation/actev_validate_system/expected_validation_results.txt
:- Add a line
actev validate-execution
with all the arguments used by the entry point to prevent any change from performers
- Add a line
- In
/diva_evaluation_cli/bin/private_src/implementation/validate_execution/score.sh
:- Modify the
Actev Scorer
call to use-V
with the right arguments according to the use of the--score
flag
- Modify the
- In
/doc/commands/cli_commands/actev_validate_execution.md
:- Add the
--score
flag in the table and in the example line
- Add the
Notes
You may find examples of activity-index, file-index, and output file in diva_evaluation_cli/container_outputs
Edited by Maxime Hubert