Commit 7a89e2a2 authored by David Thompson's avatar David Thompson Committed by Kitware Robot
Browse files

Merge topic 'test-build-failures'

2a0c02e5

 Add a CMake testing macro for expected build failures.

Acked-by: Kitware Robot's avatarKitware Robot <kwrobot@kitware.com>
Acked-by: T.J. Corona's avatarT.J. Corona <tj.corona@kitware.com>
Merge-request: !2036
parents 1090c7e2 2a0c02e5
......@@ -61,3 +61,70 @@ function(smtk_unit_tests)
endforeach(test)
endif (SMTK_ENABLE_TESTING)
endfunction(smtk_unit_tests)
# Add tests for code that is expected to fail to build.
#
# For each <test_source_file>, the compiler will be invoked <number_of_failures> times.
# Each time, the macro SMTK_FAILURE_INDEX will be defined as a different integer,
# starting with 1 and ending with <number_of_failures>.
# Thus, the same source code can validate that failure occurs in several different ways.
#
# This macro was inspired by
# https://stackoverflow.com/questions/30155619/expected-build-failure-tests-in-cmake
#
# smtk_build_failure_tests(
# LABEL <prefix for all build-failure tests>
# TESTS <test_source_file> <number_of_failures> [<test_source_file> <number_of_failures> ...]
# LIBRARIES <dependent_library_list>
# )
function(smtk_build_failure_tests)
set(options)
set(oneValueArgs)
set(multiValueArgs LABEL TESTS LIBRARIES)
cmake_parse_arguments(SMTK_bft
"${options}" "${oneValueArgs}" "${multiValueArgs}"
${ARGN}
)
list(LENGTH SMTK_bft_TESTS num_entries)
if(NOT ${num_entries})
return()
endif()
if (SMTK_ENABLE_TESTING)
smtk_get_kit_name(kit)
set(test_prog BuildFailure_${kit})
math(EXPR num_sources "${num_entries} / 2")
foreach(source_idx RANGE 0 ${num_sources} 2)
math(EXPR idx_file "2 * ${source_idx}")
math(EXPR idx_count "2 * ${source_idx} + 1")
list(GET SMTK_bft_TESTS ${idx_file} test_src)
list(GET SMTK_bft_TESTS ${idx_count} test_count)
math(EXPR trange "${test_count} - 1")
get_filename_component(thandle "${test_src}" NAME_WE)
foreach(attempt RANGE 0 ${trange})
set(tname "${test_prog}_${thandle}_${attempt}")
add_executable(${tname} ${test_src})
set_target_properties(${tname} PROPERTIES EXCLUDE_FROM_ALL TRUE EXCLUDE_FROM_DEFAULT_BUILD TRUE)
target_link_libraries(${tname} LINK_PRIVATE ${SMTK_bft_LIBRARIES})
target_compile_definitions(${tname} PRIVATE "-DSMTK_FAILURE_INDEX=${attempt}")
target_include_directories(${tname}
PRIVATE
${CMAKE_CURRENT_BINARY_DIR}
${MOAB_INCLUDE_DIRS}
${VTK_INCLUDE_DIRS}
)
add_test(
NAME ${tname}
COMMAND ${CMAKE_COMMAND} --build . --target ${tname} --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
)
set_tests_properties(${tname} PROPERTIES TIMEOUT 120 WILL_FAIL TRUE)
if(SMTK_ut_LABEL)
set_tests_properties(${tname} PROPERTIES LABELS ${SMTK_ut_LABEL})
endif()
endforeach() # attempt
endforeach() # source_idx
endif (SMTK_ENABLE_TESTING)
endfunction(smtk_build_failure_tests)
# Testing Infrastructure
A new CMake macro named `smtk_build_failure_tests` is provided in `CMake/SMTKTestingMacros.cmake`.
See `doc/userguide/contributing/testing.rst` for details.
......@@ -6,6 +6,9 @@ All new functionality added to SMTK should include tests.
When you are preparing to write tests, consider the following
* Unit tests should be present to provide coverage of new classes and methods.
* Build-failure tests provide coverage for template metaprogramming by
attempting to build code that is expected to cause static assertions or
other compilation failures.
* Integration tests should be present to ensure features work in combination
with one another as intended; these tests should model how users are expected
to exercise SMTK in a typical workflow.
......@@ -17,6 +20,61 @@ When you are preparing to write tests, consider the following
if the external project's tests all succeed, then the contract test succeeds.
Otherwise, the contract test fails.
Unit tests
----------
SMTK provides a CMake macro named ``smtk_unit_tests`` that you should use to create unit tests.
This macro will create a single executable that runs tests in multiple source files;
this reduces the number of executables in SMTK and makes tests more uniform.
Because there is a single executable, you should make your test a function whose name
matches the name of your source file (e.g., ``int TestResource(int, const char* [])``)
rather than ``int main(int, const char* [])``.
The CMake macro also allows a LABEL to be assigned to each of the tests in the executable;
this label can be used during development to run a subset of tests and during integration
to identify areas related to test failures or timings in CDash.
Build-failure tests
-------------------
Build-failure tests verify that code which is expected to cause a compiler error
does actually cause an error.
A CMake macro named ``smtk_build_failure_tests`` is
provided in :file:`CMake/SMTKTestingMacros.cmake`.
This macro generates tests that succeed when they fail to compile code you provide, as a way
to improve coverage of template metaprogramming code.
You can attempt to build the same source file multiple times; each time, a compiler macro named
`SMTK_FAILURE_INDEX` is assigned an increasing integer so you can change what code is tested.
Consider this example
.. code-block:: c++
int main()
{
#if SMTK_FAILURE_INDEX < 1
static_assert(false, "Failure mode 1");
#elif SMTK_FAILURE_INDEX < 2
static_assert(false, "Failure mode 2");
//...
#endif
return 0;
}
If this file was named `simple.cxx`, you could add tests for it with
.. code-block:: cmake
smtk_build_failure_tests(
LABEL SomeLabel
TESTS
simple.cxx 2
LIBRARIES smtkCore
)
This would create 2 tests that each try to compile `simple.cxx`
with different `SMTK_FAILURE_INDEX` values (0 and 1).
Contract tests
--------------
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment