[CMake] [CTest] Using rerun-failed when tests pass the first time
My team is using CMake as our build system, and we're also using CTest for our unit tests. We wanted to speed up running tests by only running the tests that failed previously or when the binary file had changed. While trying to use --rerun-failed
, we noticed that all our tests were being run on subsequent ctest invocations, even though they had all passed on the previous step. After looking into it, we saw that the file LastTestsFailed.log
was not being created. We tried making a simple
empty one, but that didn't change anything. We then changed one of our tests to fail, then tried running ctest
with --rerun-failed
again, and on the subsequent run it only re-ran that failed test. We reverted that test back to normal, and now --rerun-failed
only runs that test, which makes sense based on the documentation.
Is this by design? We expected that if all our tests never failed on the first invocation, that subsequent runs would not cause those tests to re-run unless the binary had changed. The behaviour also seems inconsistent, since if you add an entry into an empty LastTestsFailed.log
for a non-existant test, then subsequent test runs will not run any of your tests.
Versions:
- ctest version 3.7.2
- ctest version 3.10.1
Steps to Reproduce:
- Create a basic project with one source file and two tests.
- Build the project, and run
ctest --rerun-failed
, all tests should run. - Run again; all tests should run.
- Add a failing line to one of the two tests.
- Build and run
ctest --rerun-failed
; all tests should run. - Run 'ctest --rerun-failed' again; only the failing test should run.
- Edit the
CMakeLists.txt
to remove the failing test definition. - Build and run
ctest --rerun-failed
; outputs:No tests were found!!!
Expected Behaviour:
- If no
LastTestsFailed.log
exists, should create one; if no tests failed, then this file should be empty. - On subsequent runs, if a
LastTestsFailed.log
exists, but is empty, then no tests should be run.
Attached is a sample project demonstrating this behaviour: example.zip
Commenting/uncommenting the division test will show the issue.
/Users/ford/Downloads/example/build $ ctest --rerun-failed
Test project /Users/ford/Downloads/example/build
Start 1: addition
1/3 Test #1: addition ......................... Passed 0.00 sec
Start 2: subtraction
2/3 Test #2: subtraction ...................... Passed 0.00 sec
Start 3: multiplication
3/3 Test #3: multiplication ................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.01 sec
/Users/ford/Downloads/example/build $ ctest --rerun-failed
Test project /Users/ford/Downloads/example/build
Start 1: addition
1/3 Test #1: addition ......................... Passed 0.00 sec
Start 2: subtraction
2/3 Test #2: subtraction ...................... Passed 0.00 sec
Start 3: multiplication
3/3 Test #3: multiplication ................... Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.01 sec
Adding in the division test:
/Users/ford/Downloads/example/build $ make
[ 25%] Built target test-division
[ 50%] Built target test-addition
[ 75%] Built target test-multiplication
[100%] Built target test-subtraction
/Users/ford/Downloads/example/build $ ctest --rerun-failed
Test project /Users/ford/Downloads/example/build
Start 1: addition
1/4 Test #1: addition ......................... Passed 0.00 sec
Start 2: subtraction
2/4 Test #2: subtraction ...................... Passed 0.00 sec
Start 3: multiplication
3/4 Test #3: multiplication ................... Passed 0.00 sec
Start 4: division
4/4 Test #4: division .........................***Failed 0.00 sec
75% tests passed, 1 tests failed out of 4
Total Test time (real) = 0.02 sec
The following tests FAILED:
4 - division (Failed)
Errors while running CTest
/Users/ford/Downloads/example/build $ ctest --rerun-failed
Test project /Users/ford/Downloads/example/build
Start 4: division
1/1 Test #4: division .........................***Failed 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.01 sec
The following tests FAILED:
4 - division (Failed)
Errors while running CTest
After commenting out the division test:
/Users/ford/Downloads/example/build $ cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/ford/Downloads/example/build
/Users/ford/Downloads/example/build $ make
[ 25%] Built target test-division
[ 50%] Built target test-addition
[ 75%] Built target test-multiplication
[100%] Built target test-subtraction
/Users/ford/Downloads/example/build $ ctest --rerun-failed
Test project /Users/ford/Downloads/example/build
No tests were found!!!