Commit 8f30c710 authored by Baptiste CHOCOT's avatar Baptiste CHOCOT
Browse files

Make placebo branch closer from master

parent 9ae16e9a
Pipeline #228731 passed with stage
in 24 seconds
CLI Changelog
=============
1.2.4 - 04.22.21
----------------
* Add conda support: if you are not using conda, please remove [environment.yml](environment.yml) from your CLI.
* Add localization outputs feature
* Add proposal outputs feature
* Replace merge-chunks `result-location` parameter by `system-cache-dir`
* Add *LICENSE.txt* check during validate-system: your CLI should have a *LICENSE.txt* file at its root.
* Improve CLI and ActEV_Scorer installation
* Update entry-point parsers
* Update doc
* Remove CI for now
Also earlier...
* Add train_system entry point
* Multiple minor changes
1.2.3 - 02.27.20
----------------
================
* Upgrade to python3.
* Add [pep8](https://www.python.org/dev/peps/pep-0008/) compliance for NIST code.
* Add `--debug` option to entry point `system-setup`.
......@@ -31,89 +9,87 @@ Also earlier...
* Add script `merge_dev_into_master.sh`. It pulls the lastest changes, merges `development` on `master`, creates a tag and pushes.
1.2.2 - 08.14.19
----------------
================
* Add a script example to demonstrate data download, execution of the CLI, and validation of the outputs.
* Add support for environement variables available from one step to another. (diva_evaluation_cli.bin.private_src.implementation.utils.actev_cli_environment.*)
1.2.1 - 07.24.19
----------------
================
* Add baseline outputs on ActEV-Eval-CLI-Validation-Set3
* Update baseline outputs on ActEV-Eval-CLI-Validation-Set1 and ActEV-Eval-CLI-Validation-Set2
* Bug fixes in the validation-execution command
1.2.0 - 07.01.19
----------------
* Update validate-execution command with a `--score` flag <https://gitlab.kitware.com/actev/diva_evaluation_cli/issues/11>
* Update validate-system command, and add a `--strict` flag <https://gitlab.kitware.com/actev/diva_evaluation_cli/issues/12>
1.2.0 - 07.01.19
================
* Update validate-execution command with a `--score` flag https://gitlab.kitware.com/actev/diva_evaluation_cli/issues/11
* Update validate-system command, and add a `--strict` flag https://gitlab.kitware.com/actev/diva_evaluation_cli/issues/12
* Improved documentation for each command
* New _free disk_ metric in resources monitoring
* Bug fixes
1.1.9 - 05.23.19
----------------
================
* Bug fixes
1.1.8 - 05.15.19
----------------
================
* Bug fixes
* Less constraints on the container_output validation
1.1.7 - 04.09.19
----------------
================
* Bug fixes
* Add a new validation set: ActEV-Eval-CLI-Validation-Set2
1.1.6 - 04.01.19
----------------
================
* Improve get-system 'git': add 'recursive' option to download submodules
1.1.5 - 03.04.19
----------------
================
* Remove get-system subcommand: docker
* Rename get-system 'other' subcommand into 'archive'
* Improve get-system 'archive' and 'git' subcommands
1.1.4 - 02.22.19
----------------
================
* Bug fixes
* Add a new command: actev clear-logs
1.1.3 - 02.06.19
----------------
================
* Bug fixes
1.1.2 - 11.21.18
----------------
================
* Add a new data set: ActEV Validation Set1
* Bug fixes
1.1.1 - 11.20.18
----------------
================
* Add a new feature: resource monitoring
* Add some comments in the entry points
* Bug fixes
1.1 - 11.16.18
--------------
==============
* Add a new command: actev validate-execution
* Complete documentation
* Modify installation script: add requirements installation
* Modify installation script: add requirements installation
1.0.3 - 11.09.18
----------------
================
* Add a new command: actev status
* Add documentation about actev status
......@@ -121,20 +97,21 @@ Also earlier...
* Add actions before and after command: check video files validity in experiment-init
1.0.2 - 10.26.18
----------------
================
* Fix the installation script: support python3 virtual environments
* Modify the installation section of the README
* Add a new argument to pre/post/process-chunk: --system-cache-dir
1.0.1 - 10.24.18
----------------
================
* Add a new argument to reset-chunk: --system-cache-dir
* Modify names of the arguments composed of multiple words. Example: --chunk_id becomes --chunk-id
1.0 - 10.23.18
--------------
==============
* Release the abstract CLI: available on the master branch
* Release a baseline system CLI implementation: available on the baseline_system_master branch
This is a license.
MAKE=make
check:
@(echo "** Linting **\n")
(pycodestyle test/**/*.py diva_evaluation_cli/bin/)
@(echo "** Running UnitTests **\n")
(python3 -m unittest test/test_*/*.py)
......
# ActEV CLI README
## Introduction
Introduction
============
Consult the documentation to learn more about the CLI: [introduction](doc/introduction.md)
This repository contains:
* an abstract CLI to implement on the `master` branch.
* an implementation example on the branch `baseline_system_master`.
* an abstract CLI to implement on the `master` branch
* an implementation example on the branch `baseline_system_master`.
You should fork the project and use the `master` branch in order to implement the entry point methods and get the updates.
## Test it
Test it
============
### Get the CLI
Get the CLI
-----------
**Requirements:** python > 3.7
**Requirements: python3.X**
Clone the repository:
```bash
```
git clone https://gitlab.kitware.com/actev/diva_evaluation_cli.git
```
### Install it
Install it
----------
* Go into the clone of this repository:
```bash
cd diva_evaluation_cli
```
$ cd diva_evaluation_cli
```
* Execute the following script:
```bash
diva_evaluation_cli/bin/install.sh
```
$ diva_evaluation_cli/bin/install.sh
```
See `diva_evaluation_cli/bin/install.sh -h` for more options. If using `--all`, make sure to install the submodules before.
N.B.: We recommend to set the `-c` and `-s` options to what will be used as `system_cache_dir` by the entry_points ; otherwise, some operations like `validate-execution` will likely fail.
### Test the installation
Test the installation
---------------------
Run the following command:
```bash
actev
```
$ actev
```
You should be able to see the available subcommands.
## Usage
### Command line
Usage
=====
Command line
------------
Run the following command to obtain help with the CLI:
```bash
actev -h
```
$ actev -h
```
### Documentation
Documentation
-------------
Consult the documentation to have information: [CLI](doc/cli_commands/index.md)
## Fork it and develop your own implementation
### Fork it
Fork it and develop your own implementation
===========================================
Fork it
-------
Click on the “Fork” button to make a copy of the repository in your own space and add this repository as a remote upstream to get the latest updates
```bash
git remote add upstream https://gitlab.kitware.com/actev/diva_evaluation_cli.git
```
$ git remote add upstream https://gitlab.kitware.com/actev/diva_evaluation_cli.git
```
### Update it
Update it
---------
Execute the following commands to update it:
```bash
git fetch upstream
git checkout master
git rebase upstream/master
```
$ git fetch upstream
$ git checkout master
$ git rebase upstream/master
```
### Develop it
The CLI components are included in the `bin` directory, you do not have to modify it. To add your code, simply implement the methods in `src/entry_points`.
We suggest you to call your scripts in the entry point methods and store them in `src`.
More information about the development and the update of the CLI here: [development](doc/development.md).
Develop it
----------
:warning: Changes in release 1.2.4:
The CLI components are included in the `bin` directory, you do not have to modify it.
To add your code, you simply have to implement the methods in `src/entry_points`.
* Content of `/bin` from **your** CLI will be ignored. For example, if you used to edit `/bin/install.sh`, your submissions will likely fail from now. The best way is to move this kind of files/scripts into entry-points `actev-system-setup` and/or `actev-experiment-init`.
* CLI installation now supports [conda](https://docs.conda.io/en/latest/) environments. This is now the default way for installing the CLI dependencies, **however if you wish to stick with `pip`, you will have to remove `environment.yml` from your CLI.**
* Entry-point `validate-system` now requires the existence of a `LICENSE.txt` file at the root of your project.
We suggest you to call your scripts in the entry point methods and store them in `src`.
More information about the development and the update of the CLI here: [development](doc/development.md)
## Contact
Contact
=======
* diva-nist@nist.gov
__version__ = '1.2.4'
__version__ = '1.2.2'
......@@ -73,10 +73,7 @@ if [[ -f "environment.yml" ]]; then
eval "$conda_script"
conda activate $cli_env
fi
actev -h &> /dev/null
if [[ "$?" -ne 0 ]]; then
python3 setup.py develop --prefix $cli_env
fi
python3 setup.py develop --prefix $cli_env
# Exporting scorer env directory to the CLI shell env
cd "$(dirname "$0")"
......
......@@ -2,7 +2,6 @@
This file should not be modified.
"""
import os
......
......@@ -17,11 +17,11 @@ def entry_point(file_index, activity_index, chunks, nb_videos_per_chunk,
file_index (str): Path to file index json file for test set
activity_index (str): Path to activity index json file for test set
chunks (str): Path to chunks json file
nb_videos_per_chunk (int): Number of videos in the chunk
video_location (str): Path to videos content
system_cache_dir (str): Path to system cache directory
output_file (str): Path to merge chunks command result
chunks_result (str): Path to chunks json file after merge chunks
nb-video-per-chunk (int): Number of videos in the chunk
video-location (str): Path to videos content
system-cache-dir (str): Path to system cache directory
output-file (str): Path to merge chunks command result
chunk_result (str): Path to chunks json file after merge chunks
execution
config_file (str, optional): Path to config file
......
......@@ -2,7 +2,6 @@
This file should not be modified.
"""
import os
from diva_evaluation_cli.bin.private_src.implementation.experiment_init.\
......
......@@ -20,7 +20,6 @@ def entry_point(url, system_type, location=None, user=None, password=None,
Args:
url (str): Url to get the system
system_type (str): `git` or `archive`
location (str, optional): Path to store the system
user (str, optional): Username to access the url
password (str, optional): Password to access the url
......@@ -29,7 +28,6 @@ def entry_point(url, system_type, location=None, user=None, password=None,
installed
sha (str, optional): commit SHA or tag to checkout after cloning
name (str, optional): Directory in which the system will be saved
"""
try:
command = system_types[system_type]()
......@@ -87,7 +85,6 @@ def entry_point(url, system_type, location=None, user=None, password=None,
# go into the right directory to execute the script
path = os.path.dirname(__file__)
script = os.path.join(path, '../implementation/get_system/get/' + script)
script = "bash -e " + script
script += " " + url + \
" " + location + \
" " + user + \
......
......@@ -2,7 +2,6 @@
This file should not be modified.
"""
import os
from diva_evaluation_cli.bin.private_src.implementation.status.check_status \
......
......@@ -28,13 +28,12 @@ def entry_point(output, reference, activity_index, file_index, result, score,
Args:
output (str): Path to experiment output json file
reference (str): Path to reference json file
activity_index (str): Path to activity index json file for test set
file_index (str): Path to file index json file for test set
activity_index (str): Path to activity index json file for test set
result (str): Path to result of the ActEV scorer
score (bool): Whether scoring the system output against a reference
localization (str, optional): Path to the localization output file if
it exists
"""
# go into the right directory to execute the script
current_path = os.path.dirname(__file__)
......@@ -59,7 +58,7 @@ def entry_point(output, reference, activity_index, file_index, result, score,
if os.path.isfile(os.path.join(current_path, '../../../../environment.yml')):
# Loading dotenv and running scorer
activate_environment()
command = 'eval "$(conda shell.bash hook)"; conda activate %s; %s && \
command = 'eval "$(conda shell.bash hook)"; conda activate %s; %s; \
conda deactivate' % (os.getenv('SCORER_CONDA_ENV'), scoring_cmd)
else:
command = scoring_cmd
......
......@@ -2,7 +2,7 @@
This file should not be modified.
"""
import os
from diva_evaluation_cli.bin.private_src.implementation.validate_system.\
validate_system import validate_system
......@@ -18,6 +18,5 @@ def entry_point(strict):
Checks the structure of the directory after ActEV-system-setup is run.
Checks for expected API contents, etc.
"""
validate_system(strict)
......@@ -26,25 +26,24 @@ from diva_evaluation_cli.src.entry_points.actev_experiment_cleanup import \
entry_point as experiment_cleanup
def exec(file_index, activity_index, chunks, nb_videos_per_chunk, video_location,
def exec(file_index, activity_index, chunks, number_of_videos, video_location,
system_cache_dir, config, output, chunk_result):
"""
Args:
file_index (str): Path to file index json file for test set
activity_index (str): Path to activity index json file for test set
chunks (str): Path to chunks json file
nb_videos_per_chunk (int): Number of videos in the chunk
video_location (str): Path to videos content
system_cache_dir (str): Path to system cache directory
config (str): Path to config file
output (str): Path to merge chunks command result
nb-video-per-chunk (int): Number of videos in the chunk
video-location (str): Path to videos content
system-cache-dir (str): Path to system cache directory
output-file (str): Path to merge chunks command result
chunk_result (str): Path to chunks json file after merge chunks
execution
config_file (str, optional): Path to config file
"""
chunks_list = []
design_chunks(file_index, activity_index, chunks, nb_videos_per_chunk)
design_chunks(file_index, activity_index, chunks, number_of_videos)
try:
experiment_init(file_index, activity_index, chunks, video_location,
......
......@@ -111,7 +111,7 @@
]
}
},
"minProperties": 1,
"minProperties": 2,
"additionalProperties": false
}
},
......@@ -142,15 +142,11 @@
"type": "object",
"patternProperties": {
"^[A-Za-z0-9_\\-\\.]+$": {
"type": "object",
"properties": {
"status": {
"type": "string",
"enum": ["success", "fail"]
},
"message": {
"type": "string"
}
"status": {
"type": "string"
},
"message": {
"type": "string"
},
"required": [
"status",
......@@ -193,7 +189,6 @@
},
"required": [
"filesProcessed",
"activities",
"processingReport"
"activities"
]
}
......@@ -10,7 +10,7 @@
"message": "free text"
},
"2018-03-14.12-20-04.12-25-04.hospital.G436.avi": {
"status": "fail",
"status": "failed",
"message": "ffmpeg exited with non-zero error code"
}
},
......
......@@ -10,7 +10,7 @@
"message": "free text"
},
"2018-03-14.12-20-04.12-25-04.hospital.G436.avi": {
"status": "fail",
"status": "failed",
"message": "ffmpeg exited with non-zero error code"
}
},
......
......@@ -10,7 +10,7 @@
"message": "free text"
},
"2018-03-14.12-20-04.12-25-04.hospital.G436.avi": {
"status": "fail",
"status": "failed",
"message": "ffmpeg exited with non-zero error code"
}
},
......
{
"Closing": {
"objectTypeMap": {
"Construction_Vehicle": "*Vehicle*",
"Vehicle": "*Vehicle*"
},
"objectTypes": [
"Construction_Vehicle",
"Person",
"Vehicle"
]
},
"Closing_Trunk": {
"objectTypeMap": {
"Construction_Vehicle": "*Vehicle*",
"Vehicle": "*Vehicle*"
},
"objectTypes": [
"Construction_Vehicle",
"Person",
"Vehicle"
]
},
"Entering": {
"objectTypeMap": {
"Construction_Vehicle": "*Vehicle*",
"Vehicle": "*Vehicle*"
},
"objectTypes": [
"Construction_Vehicle",
"Person",
"Vehicle"
]
},
"Exiting": {
"objectTypeMap": {
"Construction_Vehicle": "*Vehicle*",
"Vehicle": "*Vehicle*"
},
"objectTypes": [
"Construction_Vehicle",
"Person",
"Vehicle"
]
},
"Loading": {
"objectTypeMap": {
"Construction_Vehicle": "*Vehicle*",
"Vehicle": "*Vehicle*"
},
"objectTypes": [
"Construction_Vehicle",
"Person",
"Vehicle"