ctest: Rerunning failed tests does not work properly when tests are excluded using -E option
Description:
CTest's logic for rerunning only failed tests using --rerun-failed does not work correctly if some tests have been excluded from the initial run using the -E option.
Steps to reproduce:
Define tests x, y, z with y failing Run tests excluding x with: ctest -E x Rerun failed tests only with: ctest --rerun-failed Expected: Only test y is rerun Actual: Test x is rerun instead of y
This seems to be caused by CTest relying on test indices rather than unique IDs internally. When tests are excluded with the -E option, it changes the indices and breaks the logic for rerunning failed tests.
Critically, if the tests excluded by label instead, the behavior looks good!
Suggested Fix:
Assign each test a unique ID that persists regardless of execution order. Use these IDs internally for tracking state like failed tests to rerun. This would resolve the issue and make CTest's logic more robust.
Here is an example CMakeLists.txt file that demonstrates this issue:
cmake_minimum_required(VERSION 3.10)
project(myproject)
enable_testing()
# Test x - passes
add_test(NAME x COMMAND true)
set_tests_properties(x PROPERTIES LABELS "DISABLED")
# Test y - fails
add_test(NAME y COMMAND false)
# Test z - passes
add_test(NAME z COMMAND true)
And here is what happens you run the commands:
$ ctest -E x
Test project /home/gperez/tmp/MyProject/build
Start 1: y
1/2 Test #1: y ................................***Failed 0.00 sec
Start 2: z
2/2 Test #2: z ................................ Passed 0.00 sec
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 0.01 sec
The following tests FAILED:
1 - y (Failed)
Errors while running CTest
Output from these tests are in: /home/gperez/tmp/MyProject/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
19:57:50 Wed Jul 12 ~/tmp/MyProject/build
gperez@vmx-gpere01 [0]$ ctest --rerun-failed
Test project /home/gperez/tmp/MyProject/build
Start 1: x
1/1 Test #1: x ................................ Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.00 sec
For completeness here is an example of using the label:
20:17:26 Wed Jul 12 ~/tmp/MyProject/build
gperez@vmx-gpere01 [0]$ ctest -LE DISABLED
Test project /home/gperez/tmp/MyProject/build
Start 2: y
1/2 Test #2: y ................................***Failed 0.00 sec
Start 3: z
2/2 Test #3: z ................................ Passed 0.00 sec
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 0.01 sec
The following tests FAILED:
2 - y (Failed)
Errors while running CTest
Output from these tests are in: /home/gperez/tmp/MyProject/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
20:27:00 Wed Jul 12 ~/tmp/MyProject/build
gperez@vmx-gpere01 [0]$ ctest --rerun-failed
Test project /home/gperez/tmp/MyProject/build
Start 2: y
1/1 Test #2: y ................................***Failed 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
2 - y (Failed)
Errors while running CTest
Output from these tests are in: /home/gperez/tmp/MyProject/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
20:27:19 Wed Jul 12 ~/tmp/MyProject/build