ctest_test return code doesn't account for `REPEAT UNTIL_PASS`
VTK-m has gitlab-ci setup to re-run failed tests 3 times. On the CDash side this works as expected when re-runs occur as the dashboards reports all tests as passing, but ctest returns an error code causing the gitlab-ci harness to thing a real failure occured.
Here is the output of the job, note the list of failed jobs. ( https://gitlab.kitware.com/vtk/vtk-m/-/jobs/4882189 )
99% tests passed, 2 tests failed out of 461
Label Time Summary:
CUDA = 384.27 sec*proc (168 tests)
SERIAL = 199.66 sec*proc (163 tests)
TEST_INSTALL = 0.05 sec*proc (1 test)
Total Test time (real) = 83.02 sec
The following tests FAILED:
SetCTestConfiguration:BuildDirectory:/builds/gitlab-kitware-sciviz-ci/build
SetCTestConfiguration:SourceDirectory:/builds/gitlab-kitware-sciviz-ci
SetCTestConfigurationFromCMakeVariable:DropMethod:CTEST_DROP_METHOD
SetCTestConfiguration:DropMethod:https
SetCTestConfigurationFromCMakeVariable:DropSite:CTEST_DROP_SITE
SetCTestConfiguration:DropSite:open.cdash.org
SetCTestConfigurationFromCMakeVariable:DropLocation:CTEST_DROP_LOCATION
SetCTestConfiguration:DropLocation:/submit.php?project=VTKM
Submit files
Send to group: Nightly
SubmitURL: https://open.cdash.org/submit.php?project=VTKM
Upload file: /builds/gitlab-kitware-sciviz-ci/build/Testing/20200406-0428/Test.xml to https://open.cdash.org/submit.php?project=VTKM&FileName=centos7_gcc48___relwithdebinfo%2Bcuda%2Bturing%2B32bit_ids___20200406-0428-Nightly___XML___Test.xml&build=relwithdebinfo%2Bcuda%2Bturing%2B32bit_ids&site=centos7_gcc48&stamp=20200406-0428-Nightly&MD5=3b3132de0a49aa2c4515f37e50fb8bcc Size: 894031
Uploaded: /builds/gitlab-kitware-sciviz-ci/build/Testing/20200406-0428/Test.xml
Submission successful
CMake Error at /builds/gitlab-kitware-sciviz-ci/.gitlab/ci/ctest_test.cmake:39 (message):
Failed to test
gitlab-ci harness logic: https://gitlab.kitware.com/vtk/vtk-m/-/blob/master/.gitlab/ci/ctest_test.cmake CDash reporting everything passed: https://open.cdash.org/viewTest.php?onlypassed&buildid=6449216