# VIRAT based experiments using ACT-Detector

## Introduction
These instructions walks a users through steps for replicating activity detection and activity object detection results obtained on VIRAT dataset.

    
## Contents
1. [Prerequisites](#prerequisites)
2. [Installation](#installation)
3. [Training](#training)
4. [Testing](#testing)
5. [Evaluation](#evaluation)

## Prerequisites
1. [DIVA](https://github.com/Kitware/DIVA) 
2. [CUDA-8](https://developer.nvidia.com/cuda-80-ga2-download-archive)
3. [CUDNN-6](https://developer.nvidia.com/rdp/cudnn-archive)

#### NOTE: We use `$DIVA_ROOT` for diva install directory and `$ACT_ROOT` for root directory of ACT-detector (Directory where ACT was cloned)
## Installation

1. Clone ACT using the kitware's fork of ACT-detector to `$ACT_ROOT`
    ```Shell
    git clone https://gitlab.kitware.com/kwiver/act_detector.git $ACT_ROOT
    cd $ACT_ROOT
    git checkout act-detector
    ```
2. Build caffe associated with ACT-detector 
    ```Shell
    cd $ACT_ROOT
    mkdir -p build/release
    cd build/release
    source $DIVA_ROOT/setup_DIVA.sh
    cmake -DCMAKE_INSTALL_PATH=$DIVA_ROOT -DBLAS=Open ../../
    make -j8
    ```
3. Setup correct paths for ACT and add `$DIVA_ROOT` to set_act-detector.sh to source
   the setup_DIVA.sh script

    ```Shell
    cd $ACT_ROOT
    source setup_act-detector.sh
    ```

## Training 
1. ACT-detector uses RGB and optical flow images for training. RGB images are provided in the dataset and to extract flow image
    ```Shell
    cd $DIVA_ROOT
    ./bin/optical_flow  --video_directory={PATH to RGB images} \
            --output_directory={Output Root Directory} \
            --image_extension=png --gpu_id=0
    ```
    ##### Note: The algorithm assumes that the output root directory exists and 
    does not create it. It mimics the  directory structure of the RGB data.
2. `$ACT_ROOT/virat-act-detector-scripts/rgb_eval_1b.yml` and `$ACT_ROOT/virat-act-detector-scripts/flow_eval_1b.yml` are used to specify the configuration parameters for ACT. These include path to the dataset and path to the models and prototxts. Modify the two files appropriately based on your system setup. Additional parmeters supported by these configuration files are available in `$ACT_ROOT/virat-act-detector-scripts/exp_config.py`. 
   ##### Note: `$ACT_ROOT/virat-act-detector-scripts/exp_config.py` is used to specify default values and should not be modified. To modify the parameters use `.yml` files.

3. To generate solver, train and deploy prototxt use `$ACT_ROOT/virat-act-detector-scripts/ACT_create_prototxt.py`
    ```Shell
    # For RGB model
    python  virat-act-detector-scripts/ACT_create_prototxt.py \
          --exp virat-act-detector-scripts/rgb_actev.yml
    # For Flow model
    python virat-act-detector-scripts/ACT_create_prototxt.py \
          --exp virat-act-detector-scripts/flow_actev.yml
    ```
4. Download the RGB and FLOW5 initialization models pre-trained on ILSVRC 2012
    ```Shell
    ./models/ACT-detector/scripts/fetch_initial_models.sh
    ```

5. Train the models
    ```Shell
    # To Train RGB model
    ./build/release/tools/caffe train \
        -solver models/ACT-detector/VIRAT/actev/solver_RGB.prototxt \
              -weights models/ACT-detector/initialization_VGG_ILSVRC16_K6_RGB.caffemodel \
              -gpu 0
    # To Train Flow model
    ./build/release/tools/caffe train \
        -solver models/ACT-detector/VIRAT/actev/solver_FLOW5.prototxt \
              -weights models/ACT-detector/initialization_VGG_ILSVRC16_K6_FLOW5.caffemodel \
              -gpu 0
    ``` 

## Testing
#### Note: For Activity Detection and Activity Object Detection, we used models that were trained for 150k iterations. 
1. You can either train the model using the instructions above or download the RGB and FLOW models 
    ```Shell
    cd $ACT_ROOT/models/ACT-detector/VIRAT/actev
    # RGB model
    wget https://data.kitware.com/api/v1/item/5c38dcdb8d777f072b008a91/download \
                                          -O virat_RGB_iter_150000.caffemodel
    # Flow model
    wget https://data.kitware.com/api/v1/item/5c38dc908d777f072b008a69/download \
                                          -O virat_FLOW5_iter_150000.caffemodel
    ```
2. Test and generate results for AD
    ```Shell
    python virat-act-detector-scripts/virat_test.py \
          --exp virat-act-detector-scripts/rgb_eval_1b.yml \
                  --json_path baselineACT_1_AD.json  --model_iter 150000
    ```

3. To test ACT for AOD, you would require additional object detections you can 
   download the detections offline using the following [link]() or generate it 
   using darknet in DIVA. To generate the darknet detection using DIVA refer to 
   the following [tutorial](https://github.com/Kitware/DIVA/blob/master/doc/manuals/tutorials.rst)
  
3. Test and generate results for AOD
    ```Shell
    # Unmerged object detections
    python virat-act-detector-scripts/virat_test.py \
          --exp virat-act-detector-scripts/rgb_eval_1b.yml \
                  --json_path baselinesACT_1_AOD.json  --model_iter 150000 --eval_aod
    # Merged object detections
    python virat-act-detector-scripts/virat_test.py \
          --exp virat-act-detector-scripts/rgb_actev.yml \
            --json_path baselinesACT_1_AOD.json  --model_iter 150000 \
            --eval_aod --merge_participants
    ```


## Citing ACT-detector

If you find ACT-detector useful in your research, please cite: 

    @inproceedings{kalogeiton17iccv,
      TITLE = {Action Tubelet Detector for Spatio-Temporal Action Localization},
      AUTHOR = {Kalogeiton, Vicky and Weinzaepfel, Philippe and Ferrari, Vittorio and Schmid, Cordelia},
      YEAR = {2017},
      BOOKTITLE = {ICCV},
    }
