CTest ignores the test exit code when PASS_REGULAR_EXPRESSION is set and matched
When using the PASS_REGULAR_EXPRESSION
property on a test, if that regex is matched on the output, ctest
ignores the test exit code and will report a passed test even when the test's exit code is non-zero.
Here is a reduced test case:
cmake_minimum_required(VERSION 3.0)
project(ExitCodeVsPassRegex)
include(CTest)
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/test_script.cmake "
message(expected_output)
message(FATAL_ERROR non_zero_exit_code)
")
add_test(ExpectedFail ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_BINARY_DIR}/test_script.cmake)
add_test(ShouldFail ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_BINARY_DIR}/test_script.cmake)
set_tests_properties(ShouldFail PROPERTIES PASS_REGULAR_EXPRESSION "expected_output")
$ ctest
Test project /home/sylvain/test/cmtest/build
Start 1: ExpectedFail
1/2 Test #1: ExpectedFail .....................***Failed 0.01 sec
Start 2: ShouldFail
2/2 Test #2: ShouldFail ....................... Passed 0.01 sec
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 0.02 sec
The following tests FAILED:
1 - ExpectedFail (Failed)
Errors while running CTest