Testing.txt 5.31 KB
Newer Older
Matthew Bowman's avatar
Matthew Bowman committed
1
How to run and create tests.... 
Ken Martin's avatar
Ken Martin committed
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

In creating tests there are a few goals to strive for.

1) Increase coverage of the code to help detect defects

2) Don't mess up the tester's directories. If a test wants to write out a
   file, first make sure the directory is writable, and if it is, then write
   out the files, and then delete them. So that the directory is left in its
   original state.

3) Don't require excessive disk space or CPU time. Most tests that run on a
   512 cubed volume will work just as well on a 64 cubed volume. The 64 cubed
   volume will require less disk space for the data, less CPU time to
   process, and much less CPU time to purify. The same applies to valid
   images. Typically there isn't any need for a 600 by 600 pixel valid
   image. A 300 by 300 image will work fine and require less storage and less
   CPU time to compute the image difference.

4) Try to use existing data files. If you can create a test using a data file
   that is already present, use that data file. Don't add a new data file
   unless it is critical to test the functionality you want tested. If you do
   need to add a new data file consider its size. If it is a large data file
   can you produce a lower resolution version of it and check that in instead?

The current status is that the new tree has about 100 tests and is achieving
about 43 percent coverage. These tests require about three minutes to run on
Dan Blezek's avatar
Dan Blezek committed
28
a 1 GHz Pentium and currently require about ten megabytes of disk space for
Ken Martin's avatar
Ken Martin committed
29 30
the data files and valid images. 

Ken Martin's avatar
Ken Martin committed
31 32 33 34 35 36 37 38 39 40 41 42 43

How to run a test
--------------------------

The easiest way to run a test is using ctest which is included with
cmake. You must make sure your build of VTK was done with BUILD_TESTS set to
on when running cmake. Once your build is complete you can run tests in a few
different ways. For example; you can cd to the top of your binary tree and
run ctest. This will run every test in VTk and show you the results. If you
only want to run specific tests you can use ctest -R <regexp> and provide a
regular expression or ctest -I start,end,stride,extra ... Run ctest --help to
see a more complete listing of options.

Ken Martin's avatar
Ken Martin committed
44 45 46 47
How to create a new test
---------------------------

The easiest approach is to look at an existing test and use that as a
Dan Blezek's avatar
Dan Blezek committed
48 49
starting point. The tests are kept in Testing sub-directories under each main
package. (e.g. VTK/Graphics/Testing) In Testing there will be sub-directories
Ken Martin's avatar
Ken Martin committed
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
for each type of test (Cxx Tcl Python Java). For C++ tests you need to write
the test and then add it to the CMakeLists.txt file in Testing/Cxx. For
Tclexamples you would write a Tcl script, and then add it to the
CMakeLists.txt file in Testing/Tcl. Just follow the examples already in the
tree. Most tests take a few common arguments:

  -D /path/to/VTKData
  -V relative/path/to/valid/image
  
Tcl scripts also accept:

  -A /path/to/vtktcl/package

Typically these arguments are optional and the test will try to run without
them. If the -V option is provided the test will try to perform image
differences between the valid image and the rendered image. If the valid
image specified with -V doesn't exist it will create it.

68

Ken Martin's avatar
Ken Martin committed
69 70 71 72 73
Running a Tcl example
-----------------------

The most common way to run a Tcl example is to first set TCLLIBPATH to
/path/to/VTK/Wrapping/Tcl so that the examples can find the vtktcl
Dan Blezek's avatar
Dan Blezek committed
74
package. On UNIX you can do:
Ken Martin's avatar
Ken Martin committed
75 76 77

setenv TCLLIBPATH /home/you/VTK/Wrapping/Tcl

Ken Martin's avatar
Ken Martin committed
78 79 80
or on windows set TCLLIBPATH to c:/your/VTK/Wrapping/Tcl note that UNIX style
slashes are used on windows as well. Once TCLLIBPATH is set you can run the
example as
Ken Martin's avatar
Ken Martin committed
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97

cd VTK/Graphics/Testing/Tcl
/path/to/VTK/bin/vtk yourScript.tcl 

It will look for VTKData to be at the same level as VTK on your disk. If it
isn't you will need to provide the -D argument.

/path/to/VTK/bin/vtk yourScript.tcl -D /path/to/VTKData

to run the example as a regression test you should make the example the first
argument to VTK/Common/Testing/Tcl/rtImageTest.tcl. For example

/path/to/VTK/bin/vtk /path/to/VTK/Common/Testing/Tcl/rtImageTest.tcl yourScript.tcl -D /path/to/VTKData -V Baseline/Graphics/yourScript.png

That is a long command line but normally Dart will take care of it for you.


98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
VTK-Python tests
------------------

Writing and running VTK-Python tests is documented in
Wrapping/Python/README.txt.  There are two types of Python tests, the
old tests and the newer tests that use unit testing approach
(unittest).  The newer tests can be run with a -h or --help argument.
This provides information on the available command line options.  The
other options for the newer tests are slightly different from the Tcl
test scripts.  Please read Wrapping/Python/README.txt for more details
on the newer tests.  The older tests should have the same command line
options that the Tcl scripts have.  Test writers are encouraged to use
the new approach (unittest) for the tests.


Ken Martin's avatar
Ken Martin committed
113 114 115 116
Producing a dashboard
-------------------------

If you want to test your build of VTK you can produce a dashboard. To do this
Ken Martin's avatar
Ken Martin committed
117 118 119
you will need to have cmake installed on your system (which you need anyhow
to build VTK).  Then when you run CMake, make sure BUILD_TESTING is on. Then
you can do
Ken Martin's avatar
Ken Martin committed
120 121 122

make Experimental

Ken Martin's avatar
Ken Martin committed
123 124
on UNIX to perform a build, run the tests, and submit the result to the
dashboard. If you just want to run the tests you can do
Ken Martin's avatar
Ken Martin committed
125

126
make ExperimentalTest
Ken Martin's avatar
Ken Martin committed
127 128

on windows there are targets in the workspace for Experiemental and
129
ExperimentalTest