\section{Example Algorithm Modules}
\label{sec:modules}

\begin{figure*}[t]
  \centering
  \includegraphics[width=0.58\linewidth]{content/figures/sri1.png}
  \caption{BenthosDetect as a plugin module in VIAME. Left: pipeline configuration. Middle: BenthosDetect flow diagram. Right: configuration parameters are initialized in a user configuration file and loaded up by VIAME at run time.}
  \label{fig:bd1}
\end{figure*}

A number of initial algorithm modules have either been implemented or wrapped within the platform. There are a number of ways to accomplish this based on whether or not the submodule is hosted externally on its own website, or if all of its core code rests within VIAME. Some modules are specific to individual applications, though others are more general and can be retrained to solve novel problems. A few select examples are detailed in the following sections.

\subsection{BenthosDetect}

BenthosDetect detects and identifies fish and benthic organisms that live in and on the bottom of the ocean floor. The BenthosDetect module is its own loadable plugin in the VIAME system. The core functions of the module were written in C++ and wrapped with python by boost-python.

\begin{figure}[b]
  \centering
  \includegraphics[width=\linewidth]{content/figures/sri2.png}
  \caption{Types of sea creatures used in training: background: 0, black eelpout:1, crab:2, longnose skate:3, north pacific hakefish:4, rexsole:5, rockfish:6, sea anemone:7, seasnail:8, seaurchin:9, starfish:10, sunflowerstar:11.}
  \label{fig:bd2}
\end{figure}

BenthosDetect generates regions of interest from an optical flow segmentation\cite{zhang2016unsupervised} and object proposals (e.g., selective search~\cite{uijlings2013selective}), as shown in Figure~\ref{fig:bd1}. Object proposals are based on local spatial features and flow segmentation group features based on local optical flow magnitude and orientations. Each region is processed by a pretrained deep convolutional neural network (CNN). The best score among all types determines the type of fish detected for this ROI. The modified Non-Maximum Suppression removes overlapped regions that have less score\cite{zhang2016unsupervised}. The outputs are the detected bounding boxes and corresponding types of sea creatures per frame.

The benthic organisms used in the training were annotated based on data collected at the Monterey Bay Aquarium Research Institute. 12 types of sea creatures (~12,000 samples) were trained from an annotated 1/2 hour video with no data augmentation. The classifier was tested on annotated ROI images from a separate section of video and tested as shown in Figure~\ref{fig:bd2}. The macro-averaging performance of detection from the ROC is 0.70 from the dataset. An example of detection on a frame is shown in Figure~\ref{fig:bd3}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{content/figures/sri3.png}
  \caption{Example of benthic organisms detected by BenthosDetect. Each region of interest is classified by a pretrained network and color bounding boxes are used to show types. Green boxes indicate non-organism.}
  \label{fig:bd3}
\end{figure}

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{content/figures/scalloptk.png}
  \caption{ScallopTK detector architecture. Candidate detection locations are extracted from the input image by four techniques, which are then classified according to a combination of handcrafted and automatically learned features.}
  \label{fig:scalloptk}
\end{figure*}

\subsection{ScallopFinder}

\begin{figure}[b]
  \centering
  \includegraphics[width=\linewidth]{content/figures/lanl1.jpg}
  \caption{Detections of adult scallops, baby scallops, and sand dollars by ScallopFinder.}
  \label{fig:lanl1}
\end{figure}

ScallopFinder is an algorithm being developed that is based on exploiting geometric and spectral properties of scallops. It is currently prototyped in Matlab and does not require training. Edge pixels from a canny edge detector are used to hypothesize circles by finding best fits for edge pixel chains using a circle fitting routine. The pixel RGB values are converted to HSV values and treated as a three-dimensional distribution. Next, the substrate is used as a reference to detect objects embedded in it that are spectrally distinct from the benthic substrate’s ambience. This is carried out by computing the Mahalanobis distance of each pixel HSV value from the mean of this distribution. A multiplicative filter on the distances is applied to bias them towards higher red hue and saturation. Thus, pixels that are distinct from the dominant background, and that are redder have a higher value in a reconstituted image. The hypothesized circles from the edge detection are then sampled both on the inside and the outside with respect to the reconstituted image pixels to obtain the aggregate brightness (red-filtered Mahalanobis distance) and the disparity between its interior and exterior samples. Circles with brightness and disparity above the respective median values are chosen as candidate detections. This narrows down the objects on the benthic substrate to scallops and sand dollars. 

\subsection{ScallopTK}

The Scalable Adaptive-Localization and Laplacian Object Proposal Toolkit (ScallopTK) Detector~\cite{scalloptk} is a module which is useful as a general object detector for detecting any objects which are either blob or ellipse-like. It was created primarily to detect shellfish and address the challenges with detecting them, such as differentiating between “distractor” categories (e.g. rocks, sand dollars), live organisms, and dead organisms. For each input image, the system generates many initial candidate regions of potential shellfish, and then classifies each region using a combination of AdaBoost pre-classifiers and a convolutional neural network (CNN) applied on top of sized-normalized image chips extracted around each candidate. The optional AdaBoost pre-classifiers are applied to manually-created features, and used as a speed optimization for reducing the number candidates evaluated by the CNN to a reasonable number. This full pipeline is shown in Figure~\ref{fig:scalloptk}.

\subsection{FishRuler}

The FishRuler plugin attempts to detect fish in video in order to estimate
population abundance and size distributions for particular species of fish.
It detects fish via a gaussian mixture model (GMM) segmentation, followed by
assorted heuristics to filter detections based on whether or not they are likely
to contain just one or multiple fish. Histogram of oriented gradient
classifiers~\cite{dalal2005histograms} are used on oriented extracted image chips around
each detection to classify fish species. Any detection likely to contain
multiple fish in the same connected-component blob is not used for the final
size estimation step. The module itself is currently broken into two processes.
A detector process which outputs detections from the GMM, and a classifier process which
accepts detections alongside an input image, and produces species classification scores.

\subsection{Faster R-CNN}

Faster R-CNN~\cite{fasterrcnn} was added to the platform as one of the initial object detector examples due to its generality. Unlike the hand-made object proposal generators of BenthosDetect, ScallopFinder, and ScallopTK, Faster-RCNN attempts to learn an object proposal detector using a custom region proposal network CNN architecture. The same features generated by this network are also re-used for proposal classification. In addition to Faster-RCNN we also plan on adding other high-scoring general object detectors, such as YOLO~\cite{yolo} and SSD~\cite{ssd} to serve as baseline object detectors when encountering novel problems in the domain.
