%-*- latex -*-

\chapter{Regression Testing}
\label{chap:Testing}

\index{testing|(}

\VTKm has hundreds of regression tests built-in, to test the functionality of the entire \VTKm infrastructure on new platforms. In this chapter we will discuss how to run regression tests in \VTKm, as well as how to create new regression tests.


\begin{didyouknow}
  \VTKm's regression test infrastructure is enabled by default. If you don't need regression tests and are looking for a faster compile time, you can disable it using the CMake configuration variable described in Section~\ref{sec:ConfigureVTKm}.
\end{didyouknow}


\section{Running Regression Testing}
\label{sec:RunningRegressionTests}

This section details how to run \VTKm's regression tests. First will explore how to use \textidentifier{ctest} to run these tests. \textidentifier{ctest} is the easiest option for running regression tests, as it sets a number of required arguments to the testing infrastructure automatically. Second, we will give an overview of how to run the regression tests without using ctest, and list the primary command line arguments for doing so.


\subsection{Regression Testing Using ctest}
\label{sec:regressionTestingCTEST}

\index{testing!ctest|(}
\index{ctest|(}

The following code examples show how to run the regression tests in \VTKm using \textidentifier{ctest}. Example~\ref{ex:RunningRegressionTests} shows how to run all of the enabled regression tests in \VTKm.
\begin{blankexample}[ex:RunningRegressionTests]{Running all regression tests (Unix commands).}
cd vtkm-build
ctest
\end{blankexample}

You can get a list of all the available tests by giving \textidentifier{ctest} the \textcode{-N} option, which suppresses actually running the tests (see Example~\ref{ex:ListRegressionTests}).
\begin{blankexample}[ex:ListRegressionTests]{List all available regression tests (Unix commands).}
cd vtkm-build
ctest -N
\end{blankexample}

Tests can be selected by using the \textcode{-R} option to \textidentifier{ctest}.
The \textcode{-R} option is followed by a string or regular expression to match the names of tests to run (see Example~\ref{ex:RunningSingleRegressionTests}).
\begin{blankexample}[ex:RunningSingleRegressionTests]{Running a single regression test (Unix commands).}
cd vtkm-build
ctest -R SystemInformation
\end{blankexample}

Verbose testing output can be selected by using the \textcode{-V} option to \textidentifier{ctest}.
The \textcode{-V} options causes the tests to print the underlying run command used to launch each test, along with detailed test progression information (see Example~\ref{ex:RunningSingleRegressionTestVerbose}).
\begin{blankexample}[ex:RunningSingleRegressionTestVerbose]{Running a single regression test with verbose output (Unix commands). The verbose output will first give the exact command used to run the regression test, along with detailed test progression information.}
cd vtkm-build
ctest -R -V SystemInformation
\end{blankexample}

\index{ctest|)}
\index{testing!ctest|)}



\begin{commonerrors}
  Some of the regression tests in \VTKm use data files stored in git LFS. These files are automatically pulled when the \VTKm repository is cloned. However, if the device you are compiling on does not have git LFS installed, these unit tests will fail.
\end{commonerrors}

\subsection{Regression Testing Without ctest}
\label{sec:regressionTestingWithoutCTEST}

\index{testing!without ctest|(}

It is also possible to run \VTKm regression tests without using ctest. This can be accomplished by running individual unit test wrappers that are located in the \textfilename{$<$path/to/vtk-m/build$>$/bin} directory. These tests require specific command line options in order for tests to run correctly.

Example~\ref{ex:RunningSingleRegressionTestFullCommand} shows how to run a specific rendering test by passing in the location of the \VTKm data-dir and the baseline-dir
\begin{blankexample}[ex:RunningSingleRegressionTestFullCommand]{Running a single regression test without calling ctest (Unix commands).}
UnitTests_vtkm_rendering_testing \
  UnitTestMapperVolume \
  --data-dir=@\textit{path/to/vtk-m}@/data \
  --baseline-dir=@\textit{path/to/vtk-m}@/baseline
\end{blankexample}

\index{testing!without ctest|)}



\section{Creating Regression Tests}
\label{sec:CreatingTest}

\index{testing!creating tests|(}

This section will detail the process and expectations for new regression tests in \VTKm.

\subsection{How to Add Data to \VTKm}
\VTKm uses Git LFS for all regression test data. In order to download or add test data to \VTKm you will need to have Git LFS installed. Once installed, you will add unit test data to the \textfilename{data} directory in the \VTKm repository. Data in this directory is classified according to its type: \textfilename{structured} or \textfilename{unstructured}.

\begin{blankexample}[ex:addTestData]{Adding test data to the \VTKm repository (Unix commands).}
cd vtkm-src-dir
cd data/data/<data type>
git add <file-name>
\end{blankexample}

\fix{Add further test creation instructions here.}

\begin{comment}
Topics for this section
\begin{itemize}
\item developing a test for a specific device
\item how to get the path the data directory
\item how to get the path to the regression image directory
\item where tests are located
\item how to add them to the cmake
\item what are the mandatory components of a unit test
\item how to add data to the repo
\item general policies for unit testing (code coverage, new algs should have tests, etc.)
\item how to add images for comparison
\end{itemize}
\end{comment}

\index{testing!creating tests|)}


\index{testing|)}
