|
|
## Coverage With C++
|
|
|
|
|
|
Currently coverage is only supported on gcc compiler. To perform
|
|
|
coverage test, make sure that your code is built with debug symbols,
|
|
|
without optimization, and with special flags. These flags are:
|
|
|
|
|
|
-fprofile-arcs -ftest-coverage
|
|
|
|
|
|
Also make sure to pass these flags to C compiler, CXX compiler, and the
|
|
|
linker. For example:
|
|
|
|
|
|
CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
|
|
|
-Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
|
|
|
-Wno-deprecated -Woverloaded-virtual -Wwrite-strings -fprofile-arcs -ftest-coverage"
|
|
|
CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
|
|
|
LDFLAGS="-fprofile-arcs -ftest-coverage"
|
|
|
|
|
|
Note: Older versions of ccache will cause the coverage test to fail.
|
|
|
Ctest will report that it cannot find any coverage files and this occurs
|
|
|
because ccache loses them in the two step compile-link process. Newer
|
|
|
versions have a patch to prevent this (e.g. gentoo ebuild 2.4-rt8+).
|
|
|
|
|
|
To actually run the coverage testing, you need to have tests added with
|
|
|
add_test in your CMakeLists.txt file, then you need to run a ctest -S
|
|
|
script that calls ctest_coverage, or use one of the predefined coverage
|
|
|
dashboard targets. (Use 'make help | grep Coverage' to see those.)
|
|
|
|
|
|
[Example of running coverage
|
|
|
analysis](CTest/Coverage/Example "wikilink")
|
|
|
|
|
|
### Shared libs
|
|
|
|
|
|
You might have issues with shared libs, as reported here:
|
|
|
|
|
|
- <https://twiki.cern.ch/twiki/bin/view/SPI/HowToTestCoverageReports>
|
|
|
|
|
|
(I was not able to use that trick, on a debian/oldstable and gcc 3.3.5,
|
|
|
ref
|
|
|
[here](http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/7c71804863f408ea)).
|
|
|
|
|
|
Other refs found on the
|
|
|
net:
|
|
|
|
|
|
- <http://lists.gforge.inria.fr/pipermail/visp-commits/2006-November/000273.html>
|
|
|
- <http://gcc.gnu.org/ml/gcc-help/2005-04/msg00019.html>
|
|
|
|
|
|
Conclusion: You should use gcc 3.4 and up, since gcc-3.3.5 and before
|
|
|
have this issue.
|
|
|
|
|
|
## C/C++ Coverage with Bullseye
|
|
|
|
|
|
As of June 28th 2007 CVS CMake, there is support for the cross platform
|
|
|
coverage tool Bullseye in CMake. Bullseye is a cross platform commercial
|
|
|
tool for coverage analysis. Information about Bullseye can be found
|
|
|
here: <http://www.bullseye.com/>. You will need a license for Bullseye
|
|
|
to run it.
|
|
|
|
|
|
Since Bullseye is a branch coverage tool, the output does not exacly
|
|
|
match what Dart is expecting. So, for now it makes up a fake number for
|
|
|
lines of code covered. The number is 100 \* the number of functions in
|
|
|
the file. (assume 100 lines per function). The percent coverage is a mix
|
|
|
of branches covered and functions covered.
|
|
|
|
|
|
To use Bullseye with CTest, it is easiest to use the new format ctest
|
|
|
scripts. You need to make sure the Bullseye installation bin directory
|
|
|
is in your PATH first. This can be done by setting the PATH environment
|
|
|
variable in the ctest script. Bullseye has a tool cov01 that can turn
|
|
|
on/off the coverage tool. You run that to enable/disable the coverage.
|
|
|
It should be disabled for the configure part of a project so that
|
|
|
coverage is not done on the try/compile stuff. The environment variable
|
|
|
COVFILE points to the coverage database file used by Bullseye to store
|
|
|
the coverage information. This needs to be set in the ctest script.
|
|
|
|
|
|
Here is an example ctest -S script that runs a CMake coverage dashboard
|
|
|
with bullseye:
|
|
|
|
|
|
CMAKE_MINIMUM_REQUIRED (VERSION 2.2)
|
|
|
|
|
|
# put bullseye path first
|
|
|
SET(ENV{PATH} "c:/Program\ Files/BullseyeCoverage/bin/:${PATH}")
|
|
|
|
|
|
# set variables for script
|
|
|
SET (CTEST_SITE "discworld.kitware")
|
|
|
SET (CTEST_BUILD_NAME "Windows")
|
|
|
SET (CTEST_NOTES_FILES
|
|
|
"${CTEST_SCRIPT_DIRECTORY}/${CTEST_SCRIPT_NAME}")
|
|
|
SET (CTEST_DASHBOARD_ROOT "c:/Hoffman/My Builds/")
|
|
|
SET (CTEST_SOURCE_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake")
|
|
|
SET (CTEST_BINARY_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake-coverage")
|
|
|
# Set the output file for bullsey
|
|
|
SET (ENV{COVFILE} "${CTEST_BINARY_DIRECTORY}/CMake.cov")
|
|
|
SET (CTEST_UPDATE_COMMAND "cvs")
|
|
|
SET (CTEST_CMAKE_GENERATOR "NMake Makefiles")
|
|
|
SET (CTEST_PROJECT_NAME "CMake")
|
|
|
|
|
|
# clear the binary directory and create an initial cache
|
|
|
CTEST_EMPTY_BINARY_DIRECTORY (${CTEST_BINARY_DIRECTORY})
|
|
|
# this is the initial cache to use for the binary tree, be careful to escape
|
|
|
# any quotes inside of this string if you use it
|
|
|
FILE(WRITE "${CTEST_BINARY_DIRECTORY}/CMakeCache.txt" "
|
|
|
CTEST_TEST_CTEST:BOOL=1
|
|
|
")
|
|
|
|
|
|
# Find cov01, the bullseye tool to turn on/off coverage
|
|
|
FIND_PROGRAM(COV01 cov01)
|
|
|
IF(NOT COV01)
|
|
|
MESSAGE(FATAL_ERROR "Could not find cov01")
|
|
|
ENDIF(NOT COV01)
|
|
|
# turn off coverage for configure step
|
|
|
SET(RES 1)
|
|
|
EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
|
|
|
IF(RES)
|
|
|
MESSAGE(FATAL_ERROR "could not run cov01 -0")
|
|
|
ENDIF(RES)
|
|
|
|
|
|
# do the dashboard steps
|
|
|
CTEST_START (NightlyCoverage)
|
|
|
CTEST_UPDATE (SOURCE "${CTEST_SOURCE_DIRECTORY}")
|
|
|
CTEST_CONFIGURE (BUILD "${CTEST_BINARY_DIRECTORY}")
|
|
|
# turn on coverage for build and test
|
|
|
SET(RES 1)
|
|
|
EXECUTE_PROCESS(COMMAND ${COV01} -1 RESULT_VARIABLE RES)
|
|
|
IF(RES)
|
|
|
MESSAGE(FATAL_ERROR "could not run cov01 -1")
|
|
|
ENDIF(RES)
|
|
|
CTEST_BUILD (BUILD "${CTEST_BINARY_DIRECTORY}")
|
|
|
CTEST_TEST (BUILD "${CTEST_BINARY_DIRECTORY}")
|
|
|
CTEST_COVERAGE(BUILD "${CTEST_BINARY_DIRECTORY}")
|
|
|
CTEST_SUBMIT ()
|
|
|
EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
|
|
|
|
|
|
## Coverage With Python
|
|
|
|
|
|
### Simple case
|
|
|
|
|
|
$ cat hello.py
|
|
|
"""
|
|
|
Comment #1
|
|
|
"""
|
|
|
def untested():
|
|
|
print "untested"
|
|
|
|
|
|
"""
|
|
|
Comment #2
|
|
|
"""
|
|
|
def tested():
|
|
|
print "tested"
|
|
|
|
|
|
"""
|
|
|
Comment #3
|
|
|
"""
|
|
|
if __name__ == '__main__':
|
|
|
tested()
|
|
|
|
|
|
$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir
|
|
|
/usr/lib/python2.3 hello.py
|
|
|
|
|
|
will produce a file
|
|
|
|
|
|
hello.cover
|
|
|
|
|
|
$ cat hello.cover
|
|
|
"""
|
|
|
Comment #1
|
|
|
1: """
|
|
|
1: def untested():
|
|
|
print "untested"
|
|
|
|
|
|
"""
|
|
|
Comment #2
|
|
|
"""
|
|
|
1: def tested():
|
|
|
1: print "tested"
|
|
|
|
|
|
"""
|
|
|
Comment #3
|
|
|
"""
|
|
|
1: if __name__ == '__main__':
|
|
|
1: tested()
|
|
|
|
|
|
### Complex case
|
|
|
|
|
|
Using python code from VTK:
|
|
|
|
|
|
VTK/Wrapping/Python/vtk/test/Testing.py
|
|
|
|
|
|
$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir
|
|
|
/usr/lib/python2.3 Testing.py
|
|
|
|
|
|
will produce:
|
|
|
|
|
|
ls *.cover
|
|
|
Testing.cover vtk.__init__.cover vtk.filtering.cover vtk.graphics.cover vtk.imaging.cover vtk.io.cover vtk.rendering.cover vtk.util.__init__.cover vtk.util.vtkMethodParser.cover vtk.widgets.cover
|
|
|
vtk.__helper.cover vtk.common.cover vtk.genericfiltering.cover vtk.hybrid.cover vtk.infovis.cover vtk.parallel.cover vtk.test.BlackBox.cover vtk.util.vtkConstants.cover vtk.volumerendering.cover
|
|
|
|
|
|
Example:
|
|
|
|
|
|
- [Testing.cover.txt](/uploads/4fae9c0eecc6031ee0112b8280a67242/Testing.cover.txt)
|
|
|
- [Vtk.util.vtkMethodParser.cover.txt](/uploads/9a6670e2f6b685a3f6e0a0f5451ee633/Vtk.util.vtkMethodParser.cover.txt)
|
|
|
|
|
|
### Python Trace.py Coverage In CTest
|
|
|
|
|
|
The coverage was implemented, but there are two issues.
|
|
|
|
|
|
**Minor issue \#1**: Trace.py does not differentiate between lines that
|
|
|
are not covered and the lines that cannot possibly be covered. This
|
|
|
means files with lots of comments will have poor coverage
|
|
|
|
|
|
**Minor issue \#2**: Trace.py does not work with the compiled python
|
|
|
files (\*.pyc).
|
|
|
|
|
|
**Major issue**: Trace.py does not encode the location of the source
|
|
|
file. This means that CTest has to guess where the source file is.
|
|
|
Unfortunately the guessing can be either poor or time consuming.
|
|
|
|
|
|
**Possible solution**: Make wrapper for Trace.py and include it with
|
|
|
CTest. This wrapper could solve the lack of source file location.
|
|
|
|
|
|
----
|
|
|
This page was initially populated by conversion from its [original location](https://public.kitware.com/Wiki/CTest/Coverage) in another wiki. |