|
|
## Introduction
|
|
|
|
|
|
CTest is a powerful testing client compatible with
|
|
|
[Dart](http://public.kitware.com/Dart) Dashboard system. It is
|
|
|
distributed as a part of [CMake](http://www.cmake.org) but can be used
|
|
|
with or without CMake. This tutorial explains some background behind the
|
|
|
testing with CTest and the process of generating CTest testing files.
|
|
|
|
|
|
## CTest Overview
|
|
|
|
|
|
CTest currently uses the old Dart input configuration file. New input
|
|
|
format is being developed, but the old format will be valid in the
|
|
|
future. For legacy purposes, the input files are called
|
|
|
*DartConfiguration.tcl*, and *DartTestfile.txt*. *DartConfiguration.tcl*
|
|
|
describes the system on which tests are performed, while
|
|
|
*DartTestfile.txt* provides the list of tests.
|
|
|
|
|
|
CTest reads *DartConfiguration.tcl* and depending on the options
|
|
|
provided it performs certain stage of testing.
|
|
|
|
|
|
### Description Of The System
|
|
|
|
|
|
Even though the extension of *DartConfiguration.tcl* file is *.tcl*, it
|
|
|
is not TCL file. It is simply list of key/value pairs. Line starting
|
|
|
with *\#* are considered comments and are ignored. This file has to
|
|
|
appear at the toplevel directory of the project's build directory.
|
|
|
|
|
|
Example *DartConfiguration.tcl* file is:
|
|
|
|
|
|
SourceDirectory: /path/to/project/source/directory
|
|
|
BuildDirectory: /path/to/project/build/directory
|
|
|
CVSUpdateOptions: -d -A -P
|
|
|
Site: my-system-name
|
|
|
BuildName: build-name
|
|
|
DropSite: public.kitware.com
|
|
|
DropLocation: /cgi-bin/HTTPUploadDartFile.cgi
|
|
|
DropMethod: http
|
|
|
TriggerSite: http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl
|
|
|
NightlyStartTime: 21:00:00 EDT
|
|
|
ConfigureCommand: "/path/to/source/directory/Project/configure"
|
|
|
MakeCommand: /usr/bin/make -i
|
|
|
CVSCommand: /usr/bin/cvs
|
|
|
CoverageCommand: /usr/bin/gcov
|
|
|
TimeOut: 1500
|
|
|
|
|
|
#### DartConfiguration.tcl options
|
|
|
|
|
|
| Key | Description | Example value |
|
|
|
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
| SourceDirectory | Path to the source directory of the project. Used for CVS or SVN update. | /path/to/project/source/directory |
|
|
|
| BuildDirectory | Path to the build directory of the project. This is where Testing results will go and all testing commands will be run in this directory. | /path/to/project/build/directory |
|
|
|
| Site | Refers to the system name used to distinguish the dashboard entry. | my-system-name |
|
|
|
| BuildName | A breif description of the system and the project. | FLTK-Linux-g++334 |
|
|
|
| DropSite | *DropSite*, *DropLocation*, *DropMethod*, and *TriggerSite* describe the location where test results are submitted. For more information about CTest submission, please look at: \[\[CTest:Submission_Issues | CTest:Submission_Issues\]\]. The location specified in the example is the [Public Dashboard](http://public.kitware.com/dashboard.php?name=public) provided as a service by [Kitware](http://www.kitware.com). |
|
|
|
| DropLocation | /cgi-bin/HTTPUploadDartFile.cgi |
|
|
|
| DropMethod | http |
|
|
|
| TriggerSite | <http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl> |
|
|
|
| NightlyStartTime | A the time when Nightly dashboard starts. This is useful when doing Nightly dashboard. The Nightly dashboard will update the source directory to the state at given time. | 21:00:00 EDT |
|
|
|
| UpdateCommand | Command used to update the project during the Update stage. | /usr/bin/cvs |
|
|
|
| UpdateOptions | Additional options passed to update command | \-d -A -P |
|
|
|
| UpdateType | Type of update used. Currently only CVS and SVN are valid types | cvs |
|
|
|
| ConfigureCommand | Command run during the Configure stage | "/path/to/source/directory/Project/configure" |
|
|
|
| MakeCommand | Command run during the Build stage | /usr/bin/make -i |
|
|
|
| CoverageCommand | Command used for the Coverage testing. Currently only GCC gcov is supported. | /usr/bin/gcov |
|
|
|
| TimeOut | A maximum time the test will be allowed to run. Once test time expires, the test will be interrupted and the status will be *Timeout*. | 1500 |
|
|
|
|
|
|
### Description of Tests
|
|
|
|
|
|
For legacy purposes and for compatibility with *Dart*, the test
|
|
|
description file is called *DartTestfile.txt*. It is parsed by full
|
|
|
CMake parser, but the only CMake commands that are available are
|
|
|
*SUBDIRS* and *ADD_TEST*. Example *DartTestfile.txt* is:
|
|
|
|
|
|
ADD_TEST(testProcess-1 "/my/project/testProcess" "1")
|
|
|
ADD_TEST(testProcess-2 "/my/project/testProcess" "2")
|
|
|
ADD_TEST(testProcess-3 "/my/project/testProcess" "3")
|
|
|
ADD_TEST(testProcess-4 "/my/project/testProcess" "4")
|
|
|
ADD_TEST(testProcess-5 "/my/project/testProcess" "5")
|
|
|
ADD_TEST(testProcess-6 "/my/project/testProcess" "6")
|
|
|
|
|
|
Each of the *ADD_TEST* adds the test for CTest. The first argument is
|
|
|
the unique name of test. It should be unique, so that it can be
|
|
|
differentiated among all results. The second argument is the executable
|
|
|
run as a part of the test. This is followed by arbitrary number of
|
|
|
arguments for the executable. Test executables can be built as a part of
|
|
|
the project (via add_executable), or a system executable can be used.
|
|
|
For example, to test a *Python* feature:
|
|
|
|
|
|
ADD_TEST(testTkInter "/usr/bin/python" "-c" "import Tkinter")
|
|
|
|
|
|
It is a good practice to specify full path to the executable name, since
|
|
|
that way CTest will always pick the desired executable. For example, let
|
|
|
say the test executable name is *python* but there is a python installed
|
|
|
on the system, the CTest may pick either the *python* executable built
|
|
|
by our build process or use the installed one.
|
|
|
|
|
|
### Group Tests in Subdirectories
|
|
|
|
|
|
Project tests are usually organized into some structure based on the
|
|
|
part of the project they test. For example, if the project contains some
|
|
|
common code, IO code, processing code, and GUI code, there may be group
|
|
|
of tests for each subsection. Common way of organizing this would be in
|
|
|
subdirectories. For example, let say the project has the following
|
|
|
directory structure:
|
|
|
|
|
|
/Project
|
|
|
/Common
|
|
|
/IO
|
|
|
/Processing
|
|
|
/GUI
|
|
|
|
|
|
It is fairly easy to reproduce this structure for testing:
|
|
|
|
|
|
/Project
|
|
|
/Testing
|
|
|
/Common
|
|
|
/IO
|
|
|
/Processing
|
|
|
/GUI
|
|
|
|
|
|
CTest will not automatically search all subdirectories and find all
|
|
|
*DartTestfile.txt*. The project needs to tell it where the tests are.
|
|
|
This can be achieved using *SUBDIRS* command in *DartTestfile.txt*. An
|
|
|
example use of *SUBDIRS* command is:
|
|
|
|
|
|
SUBDIRS(Testing/Common)
|
|
|
SUBDIRS(Testing/IO)
|
|
|
SUBDIRS(Testing/Processing Testing/GUI)
|
|
|
|
|
|
The arguments to the *SUBDIRS* command are subdirectories containing the
|
|
|
tests. If this file is placed in the top of our project, CTest will
|
|
|
search for files *DartTestfile.txt* inside directories
|
|
|
/.../Project/Testing/Common, /.../Project/Testing/IO, and so on to
|
|
|
/.../Project/Testing/GUI.
|
|
|
|
|
|
## Generating Testing Files
|
|
|
|
|
|
Testing files *DartConfiguration.tcl* and all ''DartTestfile.txt' must
|
|
|
be generated in the same tree as the CTest is run. In most cases this is
|
|
|
the build tree of the project. Since the testing files contain build and
|
|
|
system specific information, they should be generated as a part of
|
|
|
testing process and not permanently part of source code.
|
|
|
|
|
|
### Using CMake
|
|
|
|
|
|
CMake has a automated way of generating CTest input files. There are
|
|
|
three pieces to performing testing:
|
|
|
|
|
|
ENABLE_TESTING()
|
|
|
|
|
|
Enables testing for the project. This command will gather all tests and
|
|
|
produce *DartTestfile.txt* files with appropriate *ADD_TEST* and
|
|
|
*SUBDIRS* commands.
|
|
|
|
|
|
INCLUDE(Dart)
|
|
|
|
|
|
Including *Dart* CMake module will generate default
|
|
|
*DartConfiguration.tcl* file, which will contain information about
|
|
|
updating, configuring, and building the project.
|
|
|
|
|
|
ADD_TEST(test1 ${EXECUTABLE_OUTPUT_PATH}/test1)
|
|
|
|
|
|
Describes test that will be put in *DartTestfile.txt*. The syntax of
|
|
|
ADD_TEST on CMake level is the same as once it is written in
|
|
|
*DartTestfile.txt*.
|
|
|
|
|
|
More details about how to generate test files with CMake is described in
|
|
|
[Testing With CTest](CMake/Testing_With_CTest "wikilink") tutorial.
|
|
|
|
|
|
### Without Using CMake
|
|
|
|
|
|
CTest is distributed together with CMake, but it can be used
|
|
|
independently. The *DartConfiguration.tcl* file can be easily generated
|
|
|
using *autoconf* or any other tool, or even written by hand. Whatever
|
|
|
the method of generating that file, the author has to make sure all the
|
|
|
necessary information is written in it. Same goes for *DartTestfile.txt*
|
|
|
files.
|
|
|
|
|
|
## Conclusion
|
|
|
|
|
|
Though we consider CMake to be a great build configuration tool, CTest
|
|
|
can be used without CMake. Because of its simplicity, it can be added to
|
|
|
any project without changing the rest of the project development. Using
|
|
|
a small number of configuration files, CTest can easily automate most if
|
|
|
not all of the commonly used testing procedures.
|
|
|
|
|
|
----
|
|
|
This page was initially populated by conversion from its [original location](https://public.kitware.com/Wiki/CMake_Generating_Testing_Files) in another wiki. |