APIs and CLIs for Independent Evaluations
=========================================

**This document is focused on `Execution CLI`.**

![execution cli](figures/execution_cli.png)

Execution CLI Design Considerations
-----------------------------------

Flexibly support many types of system designs:

* Single git repo, single container, multi-container
* Scalable for multi-node solutions in the future

Rigorous validation:

* Developer-supplied expected system output on a common data set
* Conformance tests on the Execution CLI interface components
* Status checking code written by the developer to make sure processing progress is being made

Simple parallelization model to process a test collection using the unit of a ‘Chunk’:

* A ‘chunk’ is a set of video files and activities that the user can process independently on a DIVA Node
* Systems can leverage content extraction within chunk across videos
* The design of a chunk is up to the system but should be efficiently chosen
  * Could be: single video/single activity, single video/all activities, all videos collected at T0/all activities

Fault detection and recovery:

* Simplest user model possible - chunk processing either fails or succeeds
* Compute performance on intermediate and incomplete test collection runs

Submission Information
----------------------

* System Name
* System URL and access credentials
* Supported evaluation task: AD | AOD | AOTD
* Known activities: the set of pre-trained activities
* Validation set output:
  * The file json, activity json, chunk json, and the output produce by the developer at the developer’s site.
* Compute Node Limits:
  * Activities-per-chunk – max number of activities processable on a DIVA Node
  * File-streams-per-chunk – max number of camera views per DIVA Node

Note: DIVA Node == the 4 GPU node configuration

Execution CLI Overview
----------------------

NIST implemented:

* `actev get-system` – downloads a credentialed, web-accessible content into a NIST-supplied directory name \<SYS\>.
* `actev validate-system` – checks the structure of a \<SYS\> directory after ActEV-system-setup is run. Checks for expected API executables required command line options, and existing system output for validation sets.Checks for expected API executables, required command line options, and existing system output for validation sets.
* `actev exec` – a default wrapper script that calls a team-implemented API in \<SYS\> given the system config file, file-index json, activity-index json, chunk json, video location, and system cache dir.  Captures time stamps, resource usage, etc. *Supports: Independent video/activites  videos and activities are processed independently*
* `actev validate-execution` - test the execution of the system on each validation data set provided in \<SYS> comparing the  newly generated to the the expected output and the reference.
* `actev status` – executable at any time after ActEV-experiment-init exits and before ActEV-cleanup exits.
  * Reports ‘ok’ or times out
  * Retrieves log files

Team implemented from common stub provided by NIST/Kitware so as to have a consistent command line:

* `actev system-setup` – runs any compilation/preparation steps for the system.  \<SYS\> passes ActEV-validate-system after completion. Only this step should expect un-fettered internet access. Provide option `--dev` to disable monitoring/logging.
* `actev design-chunks`\* – given a file json and activity json, produce a chunk json that is suitable for the system
* `actev experiment-init`\* – specifies the system config file, file json, activity json, chunk json, video location, system cache dir, start servers (if used), starts cluster (future functionality)
* For a given ChunkID:
  * `actev pre-process-chunk`\* – specifies the ChunkID
  * `actev process-chunk`\* – detection for ChunkID
  * `actev post-process-chunk`\* – fusion within ChunkID
  * `actev reset-chunk`\* – delete all cached information for ChunkID so that the chunk can be re-run
* `actev merge-chunks`\* – returns NIST-compliant, scorable system output for the listed chunks.  Can be called at any time to get intermediate results, minimal computation expected
* `actev experiment-cleanup`\* – close any servers, terminates cluster (future functionality), etc.
* `actev train-system`\* – an optional call where the system prepares to detect surprise activities (e.g., builds models).

\* WWW access disabled

Execution CLI Definitions: \<SYS\> directory
------------------------------------------

\<SYS\> - a directory, created to contain all filesystem files for use by the analytic. For example:

NIST executes: `actev get-system -u http://some.com/forJon -S ”/tmp/Test1”` to build the directory:

* /tmp/Test1/bin/actev-system-setup
* /tmp/Test1/bin/…
* /tmp/Test1/myDockerInfo/…

Execution CLI Definitions: Chunks
---------------------------------

* A ‘chunk’ is a set of video files and activities that the user can process independently on a DIVA Node.
* Chunks are specified in a new, 3rd input JSON file for the system

![chunks](figures/chunks.png)

Evaluation CLI Process Diagrams
-------------------------------

![process cli](figures/process_cli.png)

Independent Evaluation Process Flow
-----------------------------------

![process evaluation](figures/process_evaluation.png)
