CTest return code ambiguous: Test segfault or failure?
My automated build pipeline script that runs my unit tests works like this (only showing the part that happens after using CMake to generate binary files):
ninja all && ctest || eval true
My unit test framework is Catch2. When a test case fails, ctest
returns non-zero. When a test executable segfaults, it returns the same non-zero value.
When tests cases fail normally, want the build to continue to processing the JUnit test reports. The processing of these reports will show the failed test case, and the build will fail.
When tests crash, that is not reflected in the JUnit test reports (only the successful reporting up to the crash is seen), so the build does not fail, it just appears that code coverage went way down.
I do not let CTest's non-zero return code fail the build, because if it did, we would not process the unit test reports, which means from my build tool UI I would not get any information on the test case failures.
How can I disambiguate between unit test executable crashes from unit test case failures?