CTest is a powerful testing client compatible with
Dart Dashboard system. It is
distributed as a part of CMake but can be used
with or without CMake. This tutorial explains some background behind the
testing with CTest and the process of generating CTest testing files.
CTest currently uses the old Dart input configuration file. New input
format is being developed, but the old format will be valid in the
future. For legacy purposes, the input files are called
DartConfiguration.tcl, and DartTestfile.txt. DartConfiguration.tcl
describes the system on which tests are performed, while
DartTestfile.txt provides the list of tests.
CTest reads DartConfiguration.tcl and depending on the options
provided it performs certain stage of testing.
Description Of The System
Even though the extension of DartConfiguration.tcl file is .tcl, it
is not TCL file. It is simply list of key/value pairs. Line starting
with # are considered comments and are ignored. This file has to
appear at the toplevel directory of the project's build directory.
A the time when Nightly dashboard starts. This is useful when doing Nightly dashboard. The Nightly dashboard will update the source directory to the state at given time.
Command used to update the project during the Update stage.
Additional options passed to update command
-d -A -P
Type of update used. Currently only CVS and SVN are valid types
Command run during the Configure stage
Command run during the Build stage
Command used for the Coverage testing. Currently only GCC gcov is supported.
A maximum time the test will be allowed to run. Once test time expires, the test will be interrupted and the status will be Timeout.
Description of Tests
For legacy purposes and for compatibility with Dart, the test
description file is called DartTestfile.txt. It is parsed by full
CMake parser, but the only CMake commands that are available are
SUBDIRS and ADD_TEST. Example DartTestfile.txt is:
Each of the ADD_TEST adds the test for CTest. The first argument is
the unique name of test. It should be unique, so that it can be
differentiated among all results. The second argument is the executable
run as a part of the test. This is followed by arbitrary number of
arguments for the executable. Test executables can be built as a part of
the project (via add_executable), or a system executable can be used.
For example, to test a Python feature:
It is a good practice to specify full path to the executable name, since
that way CTest will always pick the desired executable. For example, let
say the test executable name is python but there is a python installed
on the system, the CTest may pick either the python executable built
by our build process or use the installed one.
Group Tests in Subdirectories
Project tests are usually organized into some structure based on the
part of the project they test. For example, if the project contains some
common code, IO code, processing code, and GUI code, there may be group
of tests for each subsection. Common way of organizing this would be in
subdirectories. For example, let say the project has the following
/Project /Common /IO /Processing /GUI
It is fairly easy to reproduce this structure for testing:
/Project /Testing /Common /IO /Processing /GUI
CTest will not automatically search all subdirectories and find all
DartTestfile.txt. The project needs to tell it where the tests are.
This can be achieved using SUBDIRS command in DartTestfile.txt. An
example use of SUBDIRS command is:
The arguments to the SUBDIRS command are subdirectories containing the
tests. If this file is placed in the top of our project, CTest will
search for files DartTestfile.txt inside directories
/.../Project/Testing/Common, /.../Project/Testing/IO, and so on to
Generating Testing Files
Testing files DartConfiguration.tcl and all ''DartTestfile.txt' must
be generated in the same tree as the CTest is run. In most cases this is
the build tree of the project. Since the testing files contain build and
system specific information, they should be generated as a part of
testing process and not permanently part of source code.
CMake has a automated way of generating CTest input files. There are
three pieces to performing testing:
Enables testing for the project. This command will gather all tests and
produce DartTestfile.txt files with appropriate ADD_TEST and
Including Dart CMake module will generate default
DartConfiguration.tcl file, which will contain information about
updating, configuring, and building the project.
Describes test that will be put in DartTestfile.txt. The syntax of
ADD_TEST on CMake level is the same as once it is written in
More details about how to generate test files with CMake is described in
Testing With CTest tutorial.
Without Using CMake
CTest is distributed together with CMake, but it can be used
independently. The DartConfiguration.tcl file can be easily generated
using autoconf or any other tool, or even written by hand. Whatever
the method of generating that file, the author has to make sure all the
necessary information is written in it. Same goes for DartTestfile.txt
Though we consider CMake to be a great build configuration tool, CTest
can be used without CMake. Because of its simplicity, it can be added to
any project without changing the rest of the project development. Using
a small number of configuration files, CTest can easily automate most if
not all of the commonly used testing procedures.
This page was initially populated by conversion from its original location in another wiki.