CTest: Success exit code on failed submission to CDash
Presently, there does not seem to be a robust way to detect failures when submitting test results to CDash. For example, if I run ctest -D ExperimentalSubmit
and get the following response:
<cdash version="2.6.0">
<status>ERROR</status>
<message>Not a valid project.</message>
</cdash>
then ctest will return exit code 0. However, if the remote server rejects the HTTP request altogether (e.g. if the URL isn't a CDash server), then a non-zero exit code is returned.
It seems less than ideal to me that these two failure scenarios are handled differently. Both indicate that submission failed, but the only way (as far as I can tell) to detect the error reported by the CDash response is by parsing the log file for the string "Errors occurred during submission", which is a fragile approach for identifying such a potentially critical problem.
Looking at the tail end of the cmCTestSubmitHandler::ProcessHandler
function, it seems like it would be sensible to insert a return -1
in the this->HasErrors
case:
if (!this->SubmitUsingHTTP(buildDirectory + "/Testing/" +
this->CTest->GetCurrentTag(),
files, prefix, url)) {
cmCTestLog(this->CTest, ERROR_MESSAGE,
" Problems when submitting via HTTP\n");
ofs << " Problems when submitting via HTTP\n";
return -1;
}
if (this->HasErrors) {
cmCTestLog(this->CTest, HANDLER_OUTPUT,
" Errors occurred during submission.\n");
ofs << " Errors occurred during submission.\n";
return -1; // <--- PROPOSED ADDITION
} else {
cmCTestOptionalLog(this->CTest, HANDLER_OUTPUT,
" Submission successful"
<< (this->HasWarnings ? ", with warnings." : "")
<< std::endl,
this->Quiet);
ofs << " Submission successful"
<< (this->HasWarnings ? ", with warnings." : "") << std::endl;
}
return 0;
Does this seem like a sensible change, or are there good and/or historical reasons for the current behavior? I would even be happy with a --error-on-submission-failure
flag, if necessary, a la !4258 (merged).
Thank you for your time!