LEARN Framework

This is the repository for the LEARN framework. A description of the algorithms can be found in DESCRIPTION.md.

• Getting started
  • Installation
  • Downloading the JPL datasets
  • Experiment logging with Weights & Biases
  • Running the protocol
    • Protocol configurations
    • Environment variables
• Contributing an algorithm
  • Template algorithms
  • Training and evaluation process
  • Toolset
  • Adding local tasks
    • Configuration parameters
    • Budget levels
• Examples
• Docker
• Background
  • Design of the framework
  • LEARN task description
• Deploying
• Copyright

Getting started

Installation

Create a new conda environment:

conda create -n learn python=3.7 cudatoolkit=11.1 -c pytorch -c conda-forge
conda activate learn

Then, in the project root, run the following to install the framework:

git lfs install
pip install -r requirements.txt
pip install -e .
git lfs pull

Install the requirements for individual algorithms as needed. These requirements should be found in the directory corresponding to each algorithm. For example, to install the requirements for HPT:

cd learn/algorithms/HPT/
pip install -r requirements.txt

HPT for image classification and MMDET for object detection both require mmcv-full and mmdet. These can be installed with

MMCV_WITH_OPS=1 FORCE_CUDA=1 pip install mmcv-full==1.5.3 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.10.0/index.html
pip install mmdet==2.25.0

Downloading the JPL datasets

The JPL datasets can be downloaded using the scripts/instructions in the repo at https://gitlab.lollllz.com/lwll/dataset_prep. At a minimum, image classification requires mnist and object detection requires face_detection. The datasets should be placed in a directory named external.

Experiment logging with Weights & Biases

Results are logged to Weights & Biases (W&B), a platform for logging information on experiments (including configuration data and terminal stdout/stderr) and visualizing the results. To use W&B, you must:

1. be a member of the lwll project.
2. run wandb login (the wandb package is installed with the framework).

To prevent a run from being logged with W&B, add wandb_params.run_wandb=False to the run command (see Examples. W&B is not used when running with the JPL interface.

Running the protocol

The framework is configured using Hydra. Refer to the Hydra documentation for more information.

The script that runs the system is hydra_launcher.py, which can be run as in the example below:

python hydra_launcher.py \
  domain_network_selector.params.find_source_data_method=domain_net-painting

The interface (local or jpl) can be selected via the harness option. By default, the JPL interface is used.

python hydra_launcher.py \
  domain_network_selector.params.find_source_data_method=domain_net-painting \
  wandb_params.run_wandb=False \
  harness=local

Refer to the Examples section for additional sample run commands for different types of runs.

Protocol configurations

All hydra configurations can be found in configs/hydra_config. The main Hydra config file, which contains all the default values, is located at configs/hydra_config/config.yaml.

To view all configuration options, run python hydra_launcher.py --help.

Environment variables

The environment variables used for evaluation using the JPL interface can be found in scripts/test_env.sh and scripts/test_env_all.sh. The script test_env.sh specifies a single task id bbfadb2c-c7c3-4596-b548-3dd01a6d1d2c for LWLL_TA1_PROB_TASK. Thus executing the run script for that particular task. The script test_env.sh specifies all for LWLL_TA1_PROB_TASK. This would schedule tasks on all GPUs and they would be executed in parallel across all devices.

These environment variables can be specified in the Hydra run command or can be sourced (e.g., via source test_env.sh) before executing the run command. Environment variables overwrite the corresponding values in the config. Setting these environment variables is optional if you wish to use the default values.

Contributing an algorithm

Template algorithms

To contribute an algorithm, make a copy of the folder for the template algorithm most similar to the algorithm you wish to add. Algorithms (including the template algorithms) can be found in the learn/algorithms directory. For example, to add a few-shot image classification algorithm, templateImageClassification would be most appropriate.

The templates serve as random baselines and demonstrate the inputs/outputs of the algorithms.

Training and evaluation process

You must train your network in the SelectNetworkAndDataset step. You will be provided a semi-supervised transformer to use in PyTorch dataset style. You can override the transformers to set the transforms and prepare the data for your networks. At the end of this method, ensure that all CUDA memory you have used is freed.

The second step is inference, during which the network must predict labels on the evaluation dataset. The evaluation dataset is provided. You may perform transductive learning on this data. You may also use the unlabeled test data during training. The output of this method must match the format required by the relevant JPL API metrics functions.

Toolset

The toolset is a Python dictionary that contains pointers to datasets, relevant functions, configuration parameters, and other information needed by the protocol and algorithms. Using this type of object to pass information between the protocol and algorithm keeps the framework sufficiently general and able to support a variety of algorithms.

You may add parameters required by your algorithm to toolset. Refer to the algorithm templates to see how the dictionary is used and the types of parameters it may contain.

Adding local tasks

The local interface is used to run experiments on the local machine (as opposed to the JPL interface, which submits HTTP requests to the JPL server per their LWLL API). While the JPL interface obtains configuration parameters for a task from the JPL server, the local interface searches for a JSON file in which you define the configuration parameters.

Local task configurations should be placed in the configs/local_task_configs directory. At runtime, the system reads all JSON files in this directory. A sample configuration is shown below, followed by detailed information on the config schema:

{
  "name": "ic_poolcar",
  "problem_type": "image_classification",
  "results_file": "ic_results-poolcar",
  "stages": [
  {  
    "name": "base",
    "seed_budgets": [1, 11],
    "label_budget": [50, 113],
    "dataset": "pool_car_detection"
  },
  {
    "name": "adapt",
    "seed_budgets": [1],
    "label_budget": [87, 128, 188, 275, 403, 591],
    "dataset": "xview"
  }
  ],
  "whitelist": ["imagenet_1k", "coco2014"]
}

Configuration parameters

• name: A string denoting the name of the task. This should be unique, as the system identifies tasks by their name values.
• problem_type: A string denoting the problem type, which should be one of "image_classification", "machine_translation", "object_detection", or "video_classification".
• results_file: The filestem prefix for the result filename. Run results are stored in a directory named outputs by default, containing subdirectories for specific runs. The directory for a specific run contains JSON files corresponding to the output of each checkpoint for each stage of that run. For example, if results_file is set to "ic_results-poolcar", then the resulting JSON files would be named ic_results-poolcar_base_0.json (for the first checkpoint of the base stage), ic_results-poolcar_base_1.json, and so on.
• stages: A list of JSON objects, where each element of the list corresponds to a stage that should be run. Stages are run in the order in which they are defined in this list. The fields of a stage are as follows:
  • name: The name of the stage, e.g., "base" or "adapt".
  • seed_budgets: A list of x-shot seed label budgets for the first budget levels. See the Budget levels sub-section below.
  • label_budget: A list of cumulative total numbers of labels for the remaining budget levels. See the Budget levels sub-section below.
  • dataset: The name of the target dataset for this stage. The dataset must exist in the external folder; see Downloading the JPL datasets.
• whitelist: A list of whitelisted datasets that may be used as source datasets for the task.

Budget levels

Budgets are defined in two steps: 1) seed budgets and 2) label budgets.

The seed budgets denote cumulative x-shot budgets (i.e., x labels per class). In the base stage of the sample config above, the pool_car_detection dataset contains two classes, so the the first checkpoint would contain 1 labeled example per class for a total of 2 labeled training examples (2 classes x 1 label/class = 2 labels). The next seed budget adds 10 labels (the seed budgets are cumulative: 1 + 10 = 11), i.e., 11 total labeled examples per class, for a total of 22 labels at the second checkpoint.

The label budgets are cumulative label budgets (not x-shot) that follow the seed budgets. These budgets define the active/incremental learning budgets after the x-shot seed budgets have been completed. Continuing our progression through the checkpoints, the third checkpoint for the sample base stage would have a total of 50 labels and the fourth checkpoint would have 113 labels.

In total, this sample base stage would have four checkpoints with 2, 22, 50, and 113 training labels per checkpoint, respectively. Because the label budgets are cumulative, each must be greater than the preceding budget.

Examples

Specifying the problem type:

python hydra_launcher.py problem=quick_od_test

Specifying the task:

python hydra_launcher.py \
  problem=quick_od_test \
  domain_network_selector=simple_od problem.tasks_to_run=test_lvis

Specifying a name for W&B:

python hydra_launcher.py \
  problem=quick_od_test \
  domain_network_selector=simple_od \
  problem.tasks_to_run=test_lvis \
  +wandb_params.name="QUICK TEST"

Disabling W&B:

python hydra_launcher.py \
  problem=quick_od_test \
  domain_network_selector=simple_od \
  problem.tasks_to_run=test_lvis \
  wandb_params.run_wandb=False

Changing the domain network selector:

python hydra_launcher.py \
  domain_network_selector.params.find_source_data_method=domain_net-painting \
  problem.tasks_to_run=test_cifar100 \
  +wandb_params.name="TEST_CIFAR"

Changing the image classifier for an image classification experiment:

python hydra_launcher.py \
  image_classifier.name=MME/image_classification.py \
  problem.add_dataset_to_whitelist=[imagenet_1k] \
  problem.tasks_to_run=test_mars_dataset \
  +wandb_params.name="MME Imagenet Mars"

Docker

Refer to DOCKER.md for instructions on building and running a Docker container.

Background

Design of the framework

The framework is divided into protocol and algorithms. The protocol organizes and runs the algorithms, and the algorithms do the active learning. There are four different types of algorithms for this protocol.

1. Domain/Network Selection: Selects the domain and pretrained network that define the source dataset for each task.
2. Select Algorithms: Selects which algorithms to run given a source dataset and a target dataset.
3. Query Strategy: The Active Learning Query Strategy that first selects the unlabeled data which would be most impactful for labeling.
4. Adaption Algorithm: The algorithm that creates a classification for the target task. This can be image classification, video classification, object detection, or machine translation.

The diagram below details the overall problem and the protocol architecture.

LEARN task description

A task consists of any number of stages. Each stage has two datasets: a "source dataset" and a "target dataset". Additionally, we are given a whitelist of external datasets (specific to a task) that can be used for the task. We load all of these whitelisted datasets and their associated pretrained models.

At each stage, we select the whitelisted dataset (and model) most similar to the target dataset; these are termed the "source dataset" and "source model". Next, we select the best algorithm for query strategy and adaption based on the source dataset we've selected and the target dataset. The source dataset and algorithm can be manually selected by modifying the relevant configuration parameters for the task (either in the task config file or in the run command).

For active learning, we are given seed labels for the target dataset (one per class), as well as a set of budgets for the stage. These budgets are the number of labels you may have for a given checkpoint. Once you have reached the budget, you must do an evaluation. Within each budget level, we run a query strategy algorithm to select the data we wish to label, followed by a classification algorithm (either few-shot or domain adaption). Stages are run sequentially, so once all budgets for a given stage have been exhausted, you move on to the next stage.

Datasets are broken down into "external," "development," and "evaluation" datasets. Source datasets are either a dataset from a previous stage or from the set of whitelisted external datasets. External datasets come with labels. If an external dataset is on the whitelist for a task, the dataset and any models pretrained on it can be used.

Deploying

This describes the steps necessary to build, deploy, and submit the system to the LwLL eval. The system needs to be pushed to https://gitlab.lollllz.com/learn/learn, which will then automatically trigger a build.

The gitlab repo will automatically build a docker image that will then be used for evaluation. For more information on the Docker build process, please refer to the Using Docker section above, and check out DOCKER.md

1. Assuming the code is pushed to BRANCH, make sure you are on BRANCH in your local repository. The submitted build uses tags, so we will need to add a tag. Note - this tag needs to be unique and never submitted before (you can see your previously submitted tags at https://model-evaluation.lollllz.com/responses). Let's assume this is called TAG.
2. Go to Dockerfile and edit the tag to match: ARG LEARN_TAG=TAG
3. As necessary, change the ENTRYPOINT arguments in Dockerfile to match the arguments you are using for this run. Note that the LwLL system uses environment variables as specified here: https://lollllz.com/wiki/pages/viewpage.action?spaceKey=WA&title=Evaluation+Details+and+FAQs.
4. Push the changes:

git add Dockerfile
git commit -m "MESSAGE"
git push origin BRANCH

5. Tag the last commit

git tag TAG
git push origin TAG

6. (Only needs to be run once) If you don't already have the lwll remote added (git remote -v will show the available remotes), then run the following to add the LwLL repo to your remotes. If you already have it, then skip this step.

git remote add lolllz_repo  https://gitlab.lollllz.com/learn/learn.git

6. Push to the LwLL repo.

git push lolllz_repo BRANCH
git push lolllz_repo TAG

7. You should now be able to go https://gitlab.lollllz.com/learn/learn/-/tags and see the TAG you pushed. Make sure the commit and branch are correct.
8. Upon tagging, you should see a build automatically triggered here: https://gitlab.lollllz.com/learn/learn/-/jobs. There will be 3 jobs - dmc_build, dmc_tag, dmc_test. All 3 need to complete - if any of them fail then the submitted system can't run.
9. To actually submit to the LwLL system for evaluation, go to https://model-evaluation.lollllz.com/ (may need to be on the LwLL VPN). Click Submit a Response and fill out the form.
   • Team Identifier = lwll-learn
   • Put in your email address and slack handle on the LwLL slack
   • Repository Base Url = https://gitlab.lollllz.com/learn/learn
   • Tag = TAG (variable used above)
   • Set GPU Count to how many gpus you want your system to use.
   • Select which problem type(s) your system should be run against.

Testing

Requirements for testing are included in the requirements.txt file:

pip install -r requirements.txt

Run pytest to discover local unit/doc tests and run them:

pytest

Additional coverage report information may be output, like to HTML files, by providing additional --cov-report options:

pytest --cov-report html

Copyright

This material is based on research sponsored by DARPA under agreement number FA8750-19-1-0504. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.