\section{Architecture}
\label{sec:architecture}

Two key features provided by this software, DEPTH~\footnote{\label{note1}Actual name and abbreviation hidden for double blind review.}, are common data types and a stream processing toolkit called Pipeliner~\footnotemark[\ref{note1}]. Pipeliner allows the users to configure multiple processing elements in a graphical pipeline, such as in Figure~\ref{fig:pipeline}, while the common data types are used as ``edges'' in the graph. Individual nodes within this pipeline are implemented within plugin modules, both for compartmentalization and to allow modules to be easily shared.

\subsection{Pipeline Framework}

Pipeliner implements a processes and pipes data flow application structure, and an easy way to describe and run these type of applications. There are several advantages to implementing this type of application. The main benefit for DEPTH is easily interchangeable modules enabled by using common data types. For example, if one team produces a color correction algorithm, it then can be easily integrated into another’s applications or pipelines.

The main unit of construction of these pipelines is the process. A process is a class that derives from the pipeliner base process class and implements a specific operation. Both C++ and python implementations of the base class are supported, Key portions of a process are shown on the following figure. The main feature of a process are its input and output ports that are used to transfer data. The typical process lifecycle starts with creating a new object of the desired process type. This new process is then called on its configure method and passed the set of configuration items for it to use. The process uses this configuration data to establish operating parameters. In the case of algorithm wrapping processes, these configuration options will specify which implementation to use and how it should be configured. After all processes are configured, the pipeline is created by connecting process outputs to process inputs. The process is now ready to start processing and its step method is called where it reads from the input ports, processes the data and puts the results on the output ports.

Processes are designed to implement a single well defined operation so that a solution can be created by connecting these elements. Having a fine-grained approach to breaking an overall application into smaller processes promotes reuse and sharing of the processing elements. In addition, it improves the ability to parallelize the solution. 

\begin{table}[b!]
  \centering
  \begin{tabular}{|c|}
    \hline
    \small
    \begin{lstlisting}
# Process definitions and configs
#
process input
  :: frame_list_input
  :image_list_file    input_files.txt
  :frame_time         0.3333
  :image_reader:type  ocv

process detector
  :: image_object_detector
  :detector:type      ex_fish_detector
  :detector:model1    model_file.xml
  :detector:threshold 0.20

process draw
  :: draw_detected_object_boxes
  :default_line_thickness 3

process disp
  :: view_image
  :annotate_image     true
  :pause_time         2.0
  :title              DEPTH images

# Global pipeline configs
#
config _pipeline:_edge
       :capacity      5

# Connections between processes
#
connect from input.image
        to   detector.image

connect from detector.detected_object_set
        to   draw.detected_object_set
connect from input.image
        to   draw.image

connect from input.timestamp
        to   disp.timestamp
connect from draw.image
        to   disp.image
    \end{lstlisting} \\
    \hline
  \end{tabular}
  \vspace*{2mm}
  \caption{Example simple detector pipeline config file.}
  \label{tab:pipeline}
\end{table}

\subsection{Plugin Architecure}

The DEPTH platform is built on the concept of dynamically loadable plugins. A plugin is physically a shared library (DLL) that contains an implementation of software components. The DEPTH toolkit supports two types of plugins, algorithms and processes. These plugins are discovered at run time and added to their respective internal registries. The locations searched for plugins can be easily customized for any desired installation layout. The advantages from a software engineering perspective are reduced coupling between software components. For example, a plugin could be built that uses OpenCV without introducing that dependency into the core DEPTH libraries. After a plugin is created (algorithm or process), it can be individually distributed and used by others without requiring any new source code or rebuilding steps. 

Algorithms and processes are structured as a polymorphic class hierarchy. The algorithm base class is defined in DEPTH for all basic operation types, such as image filters, object detectors, and detection refining. Instantiating the concrete implementation of an algorithm is handled by a class factory mechanism built into DEPTH and controlled by user supplied configuration entries. These config entries specify the implementation type to use in addition to other implementation related parameters. Abstracting the algorithm details into a configuration file makes it easy to utilize different implementations without requiring programming expertise to modify the source code. Algorithm configurations can be directly created and modified by subject matter experts or, in the future, by using GUI tools. 

To meet the need of the research community, algorithms can be implemented in C++, Python or Matlab. The C++ implementations are implemented in a class derived from the abstract base class. Matlab implementations are a set of Matlab script files that implement the interface mediated by a C++ to Matlab adapter.

Algorithms can be instantiated and used directly in a program, or within a pipeliner process. DEPTH supplies a set of processes that support base abstract algorithm definitions. These can be considered algorithm wrapper processes which instantiate the configured algorithm then pass the process inputs to the algorithm.  Algorithm outputs are passed downstream to the next process. This approach provides algorithm level agility along with functional or topological agility.

\subsection{Basic Processes and Pipelines}

In addition to processes containing image processing algorithms, there are a number of utility processes in the system for performing tasks such as reading/writing object detections and imagery, drawing detections on images, displaying detections in a simple GUI, and others. An example pipeline is elaborated in Table~\ref{tab:pipeline} which creates an object detector alongside a few helper processes. Contained in this file is the definition of all processes in the pipeline, the data-flow between processes, and any configuration values that each process requires.

\subsection{Third Party Library Support and Build System}

Cross-platform build support for most operating systems (Windows, Mac, Linux) is provided via using CMake~\cite{cmake} instead of only a single-platform build system. Many leading open-source vision libraries involving C/C++ also currently use CMake~\cite{opencv,caffe}. Within the platform are multiple enable flags to turn on support for different third party libraries, such as OpenCV, Python and Matlab. Every individual plugin in the system also contains a corresponding enable flag, in order to compartmentalize code and allow users to only build what they need. When a certain enable flag is set, all dependencies for the plugin are also automatically turned on and built internally, in order to use as little system packages as possible and simultaneously require few dependencies when building the platform on a fresh system. Alternatively this feature can be turned off, if developers wish to build or install all dependency packages themselves externally.
