ctest: Add support for generating xUnit/JUnit XML output

changed the description

@nocnokneo see !2499 (merged) for work on a json output format. The work is done but the MR is closed temporarily while some related infrastructure is built on which it will be rebased.

Once that is done then similar code could be added for junit xml.

@brad.king Thanks, I'll take a look at that MR

mentioned in merge request !2499 (merged)

This would make Jenkins integration much simpler.

There are kind of converters exists, since steps are quite simple:

decode/uncompress compressed chunks
process with xsltproc

You can find working scripts as a part of projects, like this: https://github.com/manticoresoftware/manticoresearch/tree/master/misc/junit

Apart the files in the folder also couple of settings used. First is specific section CTestConfig.cmake

...
set(CTEST_DROP_METHOD "cp")
set(CTEST_DROP_LOCATION "here")
set(CTEST_DROP_SITE_CDASH TRUE)

second is part of ctest dashboard:

ctest_submit ()

find_program ( PHP NAMES php )
find_program ( XSLTPROC NAMES xsltproc )

file ( GLOB TESTXML "${CTEST_BINARY_DIRECTORY}/here/*Test.xml" )

execute_process (
                COMMAND ${PHP} ${CTEST_SOURCE_DIRECTORY}/misc/junit/filter.php ${TESTXML}
                COMMAND ${XSLTPROC} -o ${CTEST_BINARY_DIRECTORY}/junit_${CTEST_BUILD_CONFIGURATION}.xml ${CTEST_SOURCE_DIRECTORY}/misc/junit/ctest2junit.xsl -
)

They works in pair; submission with 'cp' ensures one known place where the files will be collected; and so, we later can process them.

But unfortunately I've found in last announce of 3.14.X that 'cp' method for submission will be no more supported... Oh, woe... Don't know will it be totally removed (so, in the case we have to stick on <3.14 release, since that method is important in our CI), or better if it will be just left 'as is'.

(actually we use 'cp' in gitlab CI for first run several test suites in parallel, collecting artifacts in predefined folder, and then in separate task we collect all of them and submit as one job using true http to cdash host. So, 'cp' artifacts is anyway available and so it can be processed in any other flavour)

They works in pair; submission with 'cp' ensures one known place where the files will be collected; and so, we later can process them.

@klirichek When ctest is run, the directory where it places its XML files can be worked out. The Testing/TAG file contains three lines (two in earlier CMake versions). The first line is the tag or timestamp of the dashboard run. It also corresponds to the directory in which you can find the XML files that would normally be uploaded to CDash and which you are needing to locate, if I'm understanding you correctly. So instead of the path ${CTEST_BINARY_DIRECTORY}/here/*Test.xml, you can use ${CTEST_BINARY_DIRECTORY}/Testing/<timestamp>/Test.xml where <timestamp> is the first line of the ${CTEST_BINARY_DIRECTORY}/Testing/TAG file for that run.

@craig.scott yep, ok, but how much such approach reliable?

Using 'cp' I just know where I place the stuff, and I can manage it. When I use internal ctest stuff like internal Testing/TAG or other files - how I can be sure it will will stay the same during, say, next 4 releases? I just want to set up CI stuff the way that it will just work during the years. Now I stick on ctest 3.13 exactly because it is the last one where method 'cp' exists.

@zackgalbreath could you provide any advice/clarification to @klirichek regarding the stability of the TAG file? It's getting a bit tangential to the main theme of this issue, but still relevant since tools that might translate the existing XML files to something suitable for xUnit, etc. would need to be able to find those XML files reliably.

This would be enormously helpful for our use of CTest from Jenkins. Using custom converters of questionable pedigree is frankly much worse. Particularly as far as reporting test outcomes, failure outputs, etc.

changed the description

This would be very helpful for our pipelines as well! Java and Python are already leveraging xUnit format - waiting on CMake to follow. Thanks so much for considering.

we are using gtest which does the job just right

Unfortunately, this doesn't really address the issue in general. Projects relying on multiple languages, or ones with full system tests (i.e., without a unit test framework like gtest) cannot benefit from the fact that gtest's output can be formatted in a desired way. Getting CTest to do it would resolve the problem in general, without reliance on specific unit test frameworks or languages.

I see that the json MR, which I understand was a prerequisite to solve this issue, is merged for months now. Is there any progress on providing the junit xml output?

We're also interested in this for Gitlab CI.

Just for reference:

https://github.com/jenkinsci/xunit-plugin/commits/master/src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl

assigned to @zackgalbreath

mentioned in merge request !6020 (merged)

ctest: Add support for generating xUnit/JUnit XML output

Child items 0

Activity