CI jobs for contract testing
VTK has a complex API that breaks occasionally without warning. In addition, behaviors sometimes change without warning. In order to make VTK developers aware of when these changes matter to "significant" projects, VTK should run contract testing for these using a successful build of VTK itself.
Projects of note (can be expanded):
-
VTK's own test suite (see Testing/External) (!7653 (merged)) -
ParaView (needs support for an external VTK to be in the repository, see paraview/paraview#14157 (closed)) -
Slicer (IIRC, needs support for a pre-built VTK)
The process would be to take a VTK CI job that has passed testing, install it to a directory and upload it as an artifact. Then each project can take this artifact, extract it into an environment where the project-under-test is cloned, built, and tested against the installed VTK.
Projects should always build, but testing is harder to quantify in addition to taking a long time. CDash results can be uploaded to both VTK's CDash instance and the main project's instance (see this file from SMTK for how to submit to multiple CDash instances). In order to subset the test suite of contract projects, tests can gain a CTest label which VTK can use to run that subset. If wanted, the full set can then be run and uploaded to the project's main CDash instance (excluding the tests already run for VTK).