ctest: Improve output when a test that is expected to fail succeeds
Given
cmake_minimum_required(VERSION 3.16)
project(foo)
enable_testing()
add_test(NAME expect_fail COMMAND echo "hi")
set_tests_properties(expect_fail PROPERTIES WILL_FAIL TRUE)
Output
$ ctest -V
ctest_will_fail/build
...
test 1
Start 1: expect_fail
1: Test command: /bin/echo "hi"
1: Test timeout computed to be: 10000000
1: hi
1/1 Test #1: expect_fail ......................***Failed 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.01 sec
The following tests FAILED:
1 - expect_fail (Failed)
Errors while running CTest
Output from these tests are in: LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
While the WILL_FAIL
property does indeed invert the result of the test, the output from ctest is not clear that the test is expected to fail, and that the test failed to pass because it actually unexpectedly succeeded (what a sentence...)
It would be nice if ctest showed some extra output to say that the test is expected to fail, and it didn't.
Edited by Brad King