ActEV Independent Evaluation CLI: Usage for Team Implementation
===============================================================

Overview
--------

The ActEV Independent Evaluation Command Line Interface (ActEV EvalCLI) defines an abstract interface for command line interaction model that is consistent for all developers while allowing freedom of underlying implementation.  Consult [introduction](introduction.md) for a more detailed description of the CLI and how it will be used.  

The abstract CLI is contained in the GIT Repo [diva_evaluation_cli](https://gitlab.kitware.com/actev/diva_evaluation_cli).  This contains a stubbed-out implementation of the CLI.  The Baseline Active Detection System (RC3D) has been adapted to the CLI as an example implementation.  It is available in the GIT repo [here](https://gitlab.kitware.com/actev/diva_evaluation_cli/tree/baseline_system_master).

The preferred workflow for the participant to implement the CLI is to GIT ‘fork’ the main branch of the abstract CLI and then implement the Entry Points.  This can be accomplished two ways: branching within the NIST GIT Repo or using a linked branch from your team’s Repo.   Contact NIST if you have questions.

We expect the EvalCLI to evolve for a short time as we finish it’s implementation.  In order to help implementers understand the intended workflow, consult the common EvalCLI change scenarios below.

Scenarios for Implementation of the ActEV Eval CLI
--------------------------------------------------

The following is a list of modification scenarios for the Eval CLI.  They are provided as a blueprint for how to incorporate changes to the CLI in existing implementations.

![Branch usage](figures/branch_usage.png)

### Scenario: NIST adds a new EvalCLI command 

NIST: 
* Add the new command in `/bin`
* Add the entry point method signature for the new command in `/src/entry_points`

Participant: 
* Merge the NIST work on its branch/repo (there is no possible conflict or break)
* Add an implementation under the new entry point method signature in `/src/entry_points`
* *pip install* the CLI, everything should work well

### Scenario: NIST adds a parameter in an existing EvalCLI command

NIST: 
* Modify the command in `/bin` by adding an argument in the CLI parser
* Add the new parameter in the right entry point method signature in `/src/entry_points`

Participant:
* Merge the NIST work on its branch/repo (there is no possible conflict or break)
* Add the new parameter within its implementation
* *pip install* the CLI, everything should work well

### Scenario: NIST modifies the type of an existing parameter (e.g. integer becomes a list of integers)

NIST: 
* Modify the command in `/bin` by modifying the argument in the CLI parser

Participant:
* Merge the NIST work on its branch/repo (there is no possible conflict but there is code breaking)
* :warning: **Need to rewrite the implementation to integrate the modification**
* *pip install* the CLI, everything should work well

### Scenario: NIST deletes a parameter 

NIST: 
* Modify the command in `/bin` by deleting the argument in the CLI parser
* Delete the new parameter in the right entry point method signature in `/src/entry_points`

Participant:
* Merge the NIST work on its branch/repo (there is no possible conflict but there is code breaking)
* :warning: **Need to rewrite the implementation to integrate the modification**
* *pip install* the CLI everything should work well

### Scenario: NIST deletes an EvalCLI command

NIST: 
* Add a warning log to encourage participant to not use the command anymore 
* Wait for a period of time to let participants remove their implementation on the entry point
* After the period of time expires, delete the command in `/bin`, delete the entry point method in `/src/entry_points`

Participant:
* Merge the NIST work on its branch/repo (possible conflict on the entry point method, need to accept NIST change if he did not remove his implementation in the entry point)
* *pip install* the CLI, everything should work well

### Scenario: Modification of /bin by a participant :warning: **THIS SHOULD BE AVOIDED**

NIST: not involved

Participant: 
* Modify the `/bin` directory (e.g. to change an argument in a command)
* *pip install* the CLI, everything should work well on its side
* Merge the NIST work on its branch/repo after an update (conflict due to the changes in `/bin`, has to accept NIST changes) 

We need a way to protect or validate `/bin` integrity: possible with *ActEV-validate-system*.

