APIs and CLIs for Independent Evaluations
=========================================

**This document is focused on `Execution CLI`.**

![execution cli](figures/execution_cli.png)

Execution CLI Design Considerations
-----------------------------------

Flexibly support many types of system designs: 
* Single git repo, single container, multi-container
* Scalable for multi-node solutions in the future

Rigorous validation:
* Developer-supplied expected system output on a common data set
* Conformance tests on the Execution CLI interface components
* Status checking code written by the developer to make sure processing progress is being made

Simple parallelization model to process a test collection using the unit of a ‘Chunk’:
* A ‘chunk’ is a set of video files and activities that the user can process independently on a DIVA Node
* Systems can leverage content extraction within chunk across videos 
* The design of a chunk is up to the system but should be efficiently chosen
  - Could be: single video/single activity, single video/all activities, all videos collected at T0/all activities

Fault detection and recovery:
* Simplest user model possible - chunk processing either fails or succeeds
* Compute performance on intermediate and incomplete test collection runs

Submission Information
----------------------

* System Name
* System URL and access credentials 
* Supported evaluation task: AD | AOD | AOTD
* Known activities: the set of pre-trained activities
* Validation set output:
  - The file json, activity json, chunk json, and the output produce by the developer at the developer’s site.
* Compute Node Limits: 
  - Activities-per-chunk – max number of activities processable on a DIVA Node
  - File-streams-per-chunk – max number of camera views per DIVA Node

*DIVA Node == the 4 GPU node configuration*

Execution CLI Overview
----------------------

NIST implemented:
* `actev get-system` – Downloads a credentialed, web-accessible content into a NIST-supplied directory name \<SYS\>. 
* `actev validate-system` – checks the structure of a \<SYS\> directory after ActEV-system-setup is run. Checks for expected API executables,  required command line options, and existing system output for validation sets.
* `actev exec` – a default wrapper script that calls a team-implemented API in \<SYS\> given the system config file, file json, activity json, chunk json, video location, system cache dir.  Captures time stamps, resource usage, etc. *Supports: Independent video/activites  videos and activities are processed independently*
* `actev validate-execution` - Test the execution of the system on each validation data set provided in <SYS> comparing the  newly generated to the the expected output and the reference.
* `actev status` – executable at any time after ActEV-experiment-init exits and before ActEV-cleanup exits.
  - Report ‘ok’ or times out
  - Retrieves log files

Team implemented from common stub provided by NIST/Kitware so as to have a consistent command line:
* `actev system-setup` – runs any compilation/preparation steps for the system.  \<SYS\> passes ActEV-validate-system after completion. Only this step should expect un-fettered internet access. Provide option `--dev` to disable monitoring/logging.
* `actev design-chunks`\* – given a file json and activity json, produce a chunk json that is suitable for the system
* `actev experiment-init`\* – specifies the system config file, file json, activity json, chunk json, video location, system cache dir, start servers (if used), starts cluster (future functionality)
* For a given ChunkID: 
  - `actev pre-process-chunk`\* – specifies the ChunkID
  - `actev process-chunk`\* – detection for ChunkID
  - `actev post-process-chunk`\* – fusion within ChunkID
  - `actev reset-chunk`\* – delete all cached information for ChunkID so that the chunk can be re-run
* `actev merge-chunks`\* – returns NIST-compliant, scorable system output for the listed chunks.  Can be called at any time to get intermediate results, minimal computation expected
* `actev experiment-cleanup`\* – close any servers, terminates cluster (future functionality), etc.

\* WWW access disabled

Execution CLI Definitions: \<SYS\> directory
------------------------------------------

\<SYS\> - a directory, created to contain all filesystem files for use by the analytic. For example:

NIST executes: `actev get-system -u http://some.com/forJon -S ”/tmp/Test1”` to build the directory:
* /tmp/Test1/bin/actev-system-setup
* /tmp/Test1/bin/…
* /tmp/Test1/myDockerInfo/…

Execution CLI Definitions: Chunks
---------------------------------

* A ‘chunk’ is a set of video files and activities that the user can process independently on a DIVA Node.
* Chunks are specified in a new, 3rd input JSON file for the system

![chunks](figures/chunks.png)

Evaluation CLI Process Diagrams
-------------------------------

![process cli](figures/process_cli.png)

Independent Evaluation Process Flow
-----------------------------------

![process evaluation](figures/process_evaluation.png)

