ctest_test return code doesn't account for `REPEAT UNTIL_PASS`
VTK-m has gitlab-ci setup to re-run failed tests 3 times. On the CDash side this works as expected when re-runs occur as the dashboards reports all tests as passing, but ctest returns an error code causing the gitlab-ci harness to thing a real failure occured.
Here is the output of the job, note the list of failed jobs. ( https://gitlab.kitware.com/vtk/vtk-m/-/jobs/4882189 )
99% tests passed, 2 tests failed out of 461 Label Time Summary: CUDA = 384.27 sec*proc (168 tests) SERIAL = 199.66 sec*proc (163 tests) TEST_INSTALL = 0.05 sec*proc (1 test) Total Test time (real) = 83.02 sec The following tests FAILED: SetCTestConfiguration:BuildDirectory:/builds/gitlab-kitware-sciviz-ci/build SetCTestConfiguration:SourceDirectory:/builds/gitlab-kitware-sciviz-ci SetCTestConfigurationFromCMakeVariable:DropMethod:CTEST_DROP_METHOD SetCTestConfiguration:DropMethod:https SetCTestConfigurationFromCMakeVariable:DropSite:CTEST_DROP_SITE SetCTestConfiguration:DropSite:open.cdash.org SetCTestConfigurationFromCMakeVariable:DropLocation:CTEST_DROP_LOCATION SetCTestConfiguration:DropLocation:/submit.php?project=VTKM Submit files Send to group: Nightly SubmitURL: https://open.cdash.org/submit.php?project=VTKM Upload file: /builds/gitlab-kitware-sciviz-ci/build/Testing/20200406-0428/Test.xml to https://open.cdash.org/submit.php?project=VTKM&FileName=centos7_gcc48___relwithdebinfo%2Bcuda%2Bturing%2B32bit_ids___20200406-0428-Nightly___XML___Test.xml&build=relwithdebinfo%2Bcuda%2Bturing%2B32bit_ids&site=centos7_gcc48&stamp=20200406-0428-Nightly&MD5=3b3132de0a49aa2c4515f37e50fb8bcc Size: 894031 Uploaded: /builds/gitlab-kitware-sciviz-ci/build/Testing/20200406-0428/Test.xml Submission successful CMake Error at /builds/gitlab-kitware-sciviz-ci/.gitlab/ci/ctest_test.cmake:39 (message): Failed to test
gitlab-ci harness logic: https://gitlab.kitware.com/vtk/vtk-m/-/blob/master/.gitlab/ci/ctest_test.cmake CDash reporting everything passed: https://open.cdash.org/viewTest.php?onlypassed&buildid=6449216