Skip to content
GitLab
Menu
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ActEV
diva_evaluation_cli
Commits
0143bd0e
Commit
0143bd0e
authored
Apr 14, 2021
by
Baptiste CHOCOT
Browse files
improve ci
parent
81fb988e
Pipeline
#223890
failed
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
.gitlab-ci.yml
View file @
0143bd0e
variables
:
# python dependencies
CONDA_ENVS
:
"
$CI_PROJECT_DIR/.conda_envs"
CLI_CONDA_ENV
:
"
$CONDA_ENVS/cli.env"
SCORER_CONDA_ENV
:
"
$CONDA_ENVS/scorer.env"
PIP_CACHE_DIR
:
"
$CI_PROJECT_DIR/.pip"
# CLIs install paths
MASTER_CLI
:
"
/tmp/master"
PERF_CLI
:
"
$CI_PROJECT_DIR"
# secrets files from nist
NIST_FILES
:
"
$CI_PROJECT_DIR/.ncache"
FILE_INDEX
:
"
$NIST_FILES/file-index.json"
ACTIVITY_INDEX
:
"
$NIST_FILES/activity-index.json"
CHUNKS_FILE
:
"
$NIST_FILES/chunks-file.json"
# system cache dir for simulation pipeline
SYSTEM_CACHE_DIR
:
"
$CI_PROJECT_DIR/.scache"
# get-system needs
CLI_URL
:
"
https://gitlab.kitware.com/actev/diva_evaluation_cli.git"
default
:
cache
:
key
:
"
$CI_COMMIT_REF_SLUG"
paths
:
-
"
$SYSTEM_CACHE_DIR"
-
"
$NIST_FILES"
-
"
$PIP_CACHE_DIR"
-
"
$CONDA_ENVS"
image
:
continuumio/anaconda3
# DEV-MERGE-TEST:
# This pipeline is used for NIST's development branches. The mergeablity with `development` and `master` branches is tested.
# SETUP:
# This pipeline will try to install the pushed CLI and merge it with NIST's.
# BASIC-TESTS:
#
# SIMULATION
#
stages
:
-
dev-merge-test
-
setup
-
merge-test
-
test
-
basic-tests
-
simulation
# Install CLI and dependencies
# STAGE DEV-MERGE-TEST
test-merge
:
stage
:
dev-merge-test
only
:
-
pushes
-
refs
- /^dev.*/
script
:
-
if [[ "$CI_COMMIT_BRANCH" -ne "development" ]]; then git checkout development ; git merge $CI_COMMIT_BRANCH; fi
-
git checkout master && git merge development
# STAGE SETUP
install
:
stage
:
setup
only
:
-
pushes
except
:
-
master
-
/^dev.*/
script
:
# need to pull master cli and install this one into the pulled one
-
python3 -m pip install pycodestyle
-
export PATH=$PATH:~/.local/bin
#- if [[ -f requirements.txt ]]; then python3 -m pip install -r requirements.txt; python3 -m pip install -e .; fi
#- if [[ -f environment.yml ]]; then mkdir $CONDA_ENVS; conda create -f environment.yml -p $CONDA_ENVS; fi
-
if [[ -f environment.yml ]]; then args="-c $CLI_CONDA_ENV -s $SCORER_CONDA_ENV"; fi
-
export PATH=$PATH:~/.local/bin
-
bash diva_evaluation_cli/bin/install.sh --all $args
# If a dev* branch, merging it on dev then master
test-merge
:
stage
:
merge-test
needs
:
[
install
]
only
:
-
pushes
-
refs
- /^dev.*/
script
:
-
if [[ "$CI_COMMIT_BRANCH" -ne "development" ]]; then git checkout development && git merge $CI_COMMIT_BRANCH; fi
-
git checkout master && git merge development
# Linting, unit tests, and system validation
test-code
:
stage
:
test
stage
:
basic-
test
s
needs
:
[
install
,
test-merge
]
only
:
-
pushes
...
...
@@ -59,14 +85,64 @@ test-code:
-
actev -h
-
actev validate-system --strict
# making sure baseline
is still working
test-
baseline
:
stage
:
test
# making sure baseline
and placebo are valid
test-
valid-systems
:
stage
:
basic-
test
s
needs
:
[
install
]
only
:
-
pushes
-
refs
- /^baseline.*/
- baseline_system_development
- /^pipeline_testing_system.*/
script
:
-
if [[ -f environment.yml ]]; then conda activate $CLI_CONDA_ENV; fi
-
actev -h
-
actev validate-system
# downloading some files for simulation
simulation-setup
:
stage
:
[
simulation
]
only
:
-
pushes
-
refs
- baseline_system_development
- /^pipeline_testing_system.*/
script
:
-
curl ... -o $FILE-INDEX
-
curl ... -o $ACTIVITY-INDEX
# testing placebos, baseline updates and performers CLI
# it emulates a few pipeline steps
test-get-system
:
stage
:
simulation
needs
:
[
test-valid-systems
,
simulation-setup
]
only
:
-
pushes
-
refs
- baseline_system_development
- /^pipeline_testing_system.*/
script
:
-
git clone https://oauth2:a38YdwbfNrksDQxo5kff@gitlab.kitware.com/actev/diva_evaluation_cli.git --recursive $MASTER_CLI
-
export PATH=$PATH:~/.local/bin
-
bash $MASTER_CLI/diva_evaluation_cli/bin/install.sh --all -c $CLI_CONDA_ENV -s $SCORER_CONDA_ENV
-
actev -h &> /dev/null
-
actev get-system git -u $CLI_URL -l $PERF_CLI -s $CI_COMMIT_BRANCH
-
exit
0
-
rm -rf $MASTER_CLI/diva_evaluation_cli/src
-
ln -s $PERF_CLI/diva_evaluation_cli/src $MASTER_CLI/diva_evaluation_cli/src
-
rm -rf $MASTER_CLI/diva_evaluation_cli/container_output
-
cp -R $PERF_CLI/diva_evaluation_cli/container_output $MASTER_CLI/diva_evaluation_cli/container_output
-
if [[ -f requirements.txt ]]; then python3 -m pip install -r $PERF_CLI/diva_evaluation_cli/requirements.txt; fi
-
if [[ -f environment.yml ]]; then bash $PERF_CLI/diva_evaluation_cli/bin/install.sh --all; fi
test-main-steps
:
stage
:
simulation
needs
:
[
test-get-system
]
only
:
-
pushes
-
refs
- baseline_system_development
- /^pipeline_testing_system.*/
script
:
-
actev experiment-init -f $FILE_INDEX -a $ACTIVITIY_INDEX -c $CHUNKS_FILE -v $VIDEO_LOCATION -s $SYSTEM_CACHE_DIR
-
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment