CTest: --output-on-failure leads to fail of the whole build target with --repeat until-pass:n when single test failed at least once.
my ctest target is like following:
add_custom_target(
RUN_SDK_TESTS
COMMAND ${CMAKE_CTEST_COMMAND} --build-config $<CONFIGURATION> --force-new-ctest-process --output-on-failure --parallel 4 --repeat until-pass:3 --tests-regex "\"^SDKT\.\""
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
)
These tests have several flakey tests, hence the need to be retried.
Legend:
O - successful
X - failed
left side of equation - tries of the same test
right side of equation - result of the whole tests run
let's consider behaviour with --repeat until-pass:3 option.
Expected behaviour:
O = O
X + O = O
X + X + O = O
X + X + X = X
Actual behaviour:
O = O
X + O = X
X + X + O = X
X + X + X = X
So, even the test passed after fail - the whole run makes error with option --output-on-failure
.
But I would like to have verbose output only from failed tests.
WORKAROUND:
replace --output-on-failure
with --verbose
It makes flood in the log but makes sense of using --repeat until-pass:3. (Expected behaviour)
Environment:
Visual Studio 2017
cmake version 3.21.2