\section{Scoring and Experimentation}
\label{sec:scoring}

\begin{figure}[t!]
  \centering
  \includegraphics[width=0.94\linewidth]{content/figures/video_player.png}
  \caption{Video viewer example. The left panel shows an entry for every spatiotemporal object track, and its corresponding category probability.}
  \label{fig:gui1}
\end{figure}

\begin{figure}[t!]
  \centering
  \includegraphics[width=1.0\linewidth]{content/figures/image_player.png}
  \caption{Image viewer example. The right hand bar has a filter for selecting which categories to show, in addition to an optional threshold on probabilities. Both computed (blue) and groundtruth (red) detections are shown in this example.}
  \label{fig:gui2}
\end{figure}

Using manual ground truth annotations, we have started comparing different variants of integrated modules within the platform. There are two tools which accomplish this, one for ROC generation and one for generating fixed scoring metrics (detection rate, false alarm rates, etc). Both of these tools can score either individual object detections, spatiotemporal object tracks, or spatiotemporal events against some groundtruth. Due to the multitude of different ways to score the same problem, the scoring tools have evolved to contain a number of input parameters over time, such as the spatial and temporal overlap criteria of computed on groundtruthed tracks. Figure~\ref{fig:rocs} shows an example ROC generated by these tools comparing several detector variants in the system for the same problem, and Table~\ref{tab:scoretracks} shows an example fixed metric output.

\begin{table}[h!]
  \centering
  \begin{tabular}{|c|}
    \hline
    \tiny
    \begin{lstlisting}
Detection-Pd: 0.791209
Detection-FA: 213
Detection-PFA: 0.515738
Frame-NFAR: not computed
Track-Pd: 0.791209
Track-FA: 213
Computed-track-PFA: 0.515738
Track-NFAR: not computed
Avg track (cont., purity ): 1.34, 1
Avg target (cont., purity ): 1.47, 0.79
Track-frame-precision: 0.5

DEPTH-Hash : "a2123cde"
    \end{lstlisting} \\
    \hline
  \end{tabular}
  \vspace*{2mm}
  \caption{Example ``score tracks'' tool output. In this case, only single-frame object detections were scored on the same scenario as in Figure~\ref{fig:rocs}, therefore the track and detection Pd (Percent detection) statistics are equivalent.}
  \label{tab:scoretracks}
\end{table}

\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\linewidth]{content/figures/roc2.png}
  \includegraphics[width=0.95\linewidth]{content/figures/roc1.png}
  \caption{Example ROCs generated by the system comparing different detectors. In this case, a 600 image test sequence containing scallops (the targets of interest in this experiment) was used. Both ROCs show the same content, though the bottom image shows logarithmic scaling of the x-axis.}
  \label{fig:rocs}
\end{figure}
