# ActEV Independent Evaluation CLI: Usage for Team Implementation

## Overview

The ActEV Independent Evaluation Command Line Interface (ActEV EvalCLI) defines an abstract interface for command line interaction model that is consistent for all developers while allowing freedom of underlying implementation.  Consult [introduction](introduction.md) for a more detailed description of the CLI and how it will be used.  

The abstract CLI is contained in the GIT Repo [diva_evaluation_cli](https://gitlab.kitware.com/actev/diva_evaluation_cli).  This contains a stubbed-out implementation of the CLI.  The Baseline Active Detection System (RC3D) has been adapted to the CLI as an example implementation.  It is available in the GIT repo [here](https://gitlab.kitware.com/actev/diva_evaluation_cli/tree/baseline_system_master).

The preferred workflow for the participant to implement the CLI is to GIT ‘fork’ the main branch of the abstract CLI and then implement the Entry Points.  This can be accomplished two ways: branching within the NIST GIT Repo or using a linked branch from your team’s Repo.   Contact NIST if you have questions.

We expect the EvalCLI to evolve for a short time as we finish it’s implementation.  In order to help implementers understand the intended workflow, consult the common EvalCLI change scenarios below.

## Python dependencies

There are now two ways to install and manage the dependencies of the ActEV CLI. The newer and default one is `conda`, and the old one is `pip`. When installing the CLI, it will consider you are using conda if the file `environment.yml` exists at the root of the project. If it doesn't, it will use `requirements.txt`.

The `/bin/install.sh` script should be enough to manage the dependencies, but you can still do it manually.

### Create the environment manually

* conda:

```bash
conda env create -f environment.yml -p <path-to-environment>
```

* pip:

```bash
[python3 -m] pip install -r requirements.txt
```

### Update the environment manually

* conda:

```bash
conda env update -f environment.yml -p <path-to-environment>
```

* pip:

```bash
python3 -m pip install -r requirements.txt --upgrade
```

## Scenarios for Implementation of the ActEV Eval CLI

The following is a list of modification scenarios for the Eval CLI.  They are provided as a blueprint for how to incorporate changes to the CLI in existing implementations.

![Branch usage](figures/branch_usage.png)

### Scenario: NIST adds a new EvalCLI command

NIST:

* Add the new command in `/bin`
* Add the entry point method signature for the new command in `/src/entry_points`

Participant:

* Merge the NIST work on its branch/repo (there is no possible conflict or break)
* Add an implementation under the new entry point method signature in `/src/entry_points`
* *pip install* or *conda env update* the CLI, everything should work well

### Scenario: NIST adds a parameter in an existing EvalCLI command

NIST:

* Modify the command in `/bin` by adding an argument in the CLI parser
* Add the new parameter in the right entry point method signature in `/src/entry_points`

Participant:

* Merge the NIST work on its branch/repo (there is no possible conflict or break)
* Add the new parameter within its implementation
* *pip install* or *conda env update* the CLI, everything should work well

### Scenario: NIST modifies the type of an existing parameter (e.g. integer becomes a list of integers)

NIST:

* Modify the command in `/bin` by modifying the argument in the CLI parser

Participant:

* Merge the NIST work on its branch/repo (there is no possible conflict but there is code breaking)
* :warning: **Need to rewrite the implementation to integrate the modification**
* *pip install* or *conda env update* the CLI, everything should work well

### Scenario: NIST deletes a parameter

NIST:

* Modify the command in `/bin` by deleting the argument in the CLI parser
* Delete the new parameter in the right entry point method signature in `/src/entry_points`

Participant:

* Merge the NIST work on its branch/repo (there is no possible conflict but there is code breaking)
* :warning: **Need to rewrite the implementation to integrate the modification**
* *pip install* or *conda env update* the CLI everything should work well

### Scenario: NIST deletes an EvalCLI command

NIST:

* Add a warning log to encourage participant to not use the command anymore
* Wait for a period of time to let participants remove their implementation on the entry point
* After the period of time expires, delete the command in `/bin`, delete the entry point method in `/src/entry_points`

Participant:

* Merge the NIST work on its branch/repo (possible conflict on the entry point method, need to accept NIST change if he did not remove his implementation in the entry point)
* *pip install* or *conda env update* the CLI, everything should work well

### Scenario: Modification of /bin by a participant :warning: **THIS SHOULD BE AVOIDED**

NIST: not involved

Participant:

* Modify the `/bin` directory (e.g. to change an argument in a command)
* *pip install* the CLI, everything should work well on its side
* Merge the NIST work on its branch/repo after an update (conflict due to the changes in `/bin`, has to accept NIST changes)

We need a way to protect or validate `/bin` integrity: possible with *ActEV-validate-system*.
