Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • D diva_evaluation_cli
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • ActEV
  • diva_evaluation_cli
  • Issues
  • #11
Closed
Open
Created Jun 03, 2019 by Alexandre Boyer@alexandreBMaintainer

actev validate-execution command: add a flag to validate/score an output

What is the current behavior?

The actev validate-execution command is currently able to validate and score an output against a reference as follows:

actev validate-execution 
-o <path to output result> 
-r <path to reference file> 
-a <path to activity> 
-f <path to file> 
-R <path to scoring result>

What is the desired behavior?

It would be great to add a flag --score to the actev validate-execution. The new default behavior would only validate the output without scoring it:

actev validate-execution 
-o <path to output result> 
-a <path to activity> 
-f <path to file> 

The command using the --score flag would validate and score the output:

actev validate-execution 
-o <path to output result> 
-r <path to reference file> 
-a <path to activity> 
-f <path to file> 
-R <path to scoring result>
--score

Changes

  • In /diva_evaluation_cli/bin/commands/actev_validate_execution.py:
    • Add an argument --score in the validate-execution parser in /bin/commands/actev-validate-execution.py
    • Remove -r --reference and -R --results` arguments from the required argument group: they are not mandatory anymore
  • In /diva_evaluation_cli/bin/private_src/entry_points/actev_validate_execution.py:
    • Add a new parameter in the function signature: score
    • Integrate the argument to the script variable that is passed to the implementation part.
  • In /diva_evaluation_cli/bin/private_src/implementation/actev_validate_system/expected_validation_results.txt:
    • Add a line actev validate-execution with all the arguments used by the entry point to prevent any change from performers
  • In /diva_evaluation_cli/bin/private_src/implementation/validate_execution/score.sh:
    • Modify the Actev Scorer call to use -V with the right arguments according to the use of the --score flag
  • In /doc/commands/cli_commands/actev_validate_execution.md:
    • Add the --score flag in the table and in the example line

Notes

You may find examples of activity-index, file-index, and output file in diva_evaluation_cli/container_outputs

Edited Jun 05, 2019 by Maxime Hubert
Assignee
Assign to
Time tracking