Many tests fail when run from Jenkins/Bamboo with DISPLAY set to good opengl-enabled display.
Hi everybody,
I'm running the vtk nightly builds using jenkins & bamboo, and I've got an inexplicable issue with many tests failing. The automated builds set DISPLAY so tests run on a good X display, with nvidia opengl- and most tests work fine, but approximately 350 tests fail for reasons I cannot explain.
When I run the same build scripts from a terminal running on the actual X display, all tests complete successfully.
I've isolated a command to run one of the failing tests, and I can replicate the problem easily, with a script containing just one test:
VTK_BUILD_DIR="/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build"
DISPLAY=:2.0 "${VTK_BUILD_DIR}/bin/vtkChartsCoreCxxTests" "TestMultipleRenderers" "-E" "25" "-D" "${VTK_BUILD_DIR}/ExternalData/Testing" "-T" "${VTK_BUILD_DIR}/Testing/Temporary" "-V" "${VTK_BUILD_DIR}/ExternalData/Charts/Core/Testing/Data/Baseline/TestMultipleRenderers.png"
Running the test from a terminal local to the X display is all good:
<DartMeasurement name="ImageError" type="numeric/double">0</DartMeasurement><DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement><DartMeasurement name="WallTime" type="numeric/double">0.0331759</DartMeasurement>
<DartMeasurement name="CPUTime" type="numeric/double">0.079182</DartMeasurement>
Running the test from a terminal inside another vncserver, but with DISPLAY set to point to the proper Xorg display, gives:
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions: GLX_EXT_visual_info GLX_EXT_visual_rating GLX_EXT_import_context GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age GLX_ARB_create_context GLX_ARB_create_context_profile GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile GLX_ARB_create_context_robustness GLX_NV_delay_before_swap GLX_EXT_stereo_tree GLX_EXT_libglvnd GLX_ARB_context_flush_control GLX_NV_robustness_video_memory_purge GLX_ARB_multisample GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_EXT_framebuffer_sRGB GLX_NV_copy_image
client glx vendor string: NVIDIA Corporation
client glx version string: 1.4
glx extensions: GLX_EXT_visual_info GLX_EXT_visual_rating GLX_EXT_import_context GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age GLX_ARB_create_context GLX_ARB_create_context_profile GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile GLX_ARB_create_context_robustness GLX_NV_delay_before_swap GLX_EXT_stereo_tree GLX_ARB_context_flush_control GLX_NV_robustness_video_memory_purge GLX_ARB_multisample GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_EXT_framebuffer_sRGB GLX_NV_copy_image GLX_ARB_get_proc_address
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 960/PCIe/SSE2
OpenGL version string: 4.5.0 NVIDIA 384.98
OpenGL extensions:
...
lots of extensions
...
X Extensions: Generic Event Extension, SHAPE, MIT-SHM, XInputExtension, XTEST, BIG-REQUESTS, SYNC, XKEYBOARD, XC-MISC, XFIXES, RENDER, RANDR, XINERAMA, Composite, DAMAGE, MIT-SCREEN-SAVER, DOUBLE-BUFFER, RECORD, DPMS, Present, X-Resource, XVideo, XFree86-VidModeExtension, XFree86-DGA, DRI2, GLX, NV-GLX, NV-CONTROL, XINERAMA
<DartMeasurement name="ImageError" type="numeric/double">244344</DartMeasurement><DartMeasurement name="BaselineImage" type="numeric/integer">1</DartMeasurement><DartMeasurementFile name="TestImage" type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/Testing/Temporary/TestMultipleRenderers.png</DartMeasurementFile>
Failed Image Test ( TestMultipleRenderers.png ) : 244344
<DartMeasurementFile name="DifferenceImage" type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/Testing/Temporary/TestMultipleRenderers.diff.png</DartMeasurementFile><DartMeasurementFile name="ValidImage" type="image/png">/data-ssd/data/development/src/vtk/build/jenkins/dashboard/nightly/1/vtk-build/ExternalData/Charts/Core/Testing/Data/Baseline/TestMultipleRenderers_1.png</DartMeasurementFile><DartMeasurement name="WallTime" type="numeric/double">0.695937</DartMeasurement>
<DartMeasurement name="CPUTime" type="numeric/double">1.87324</DartMeasurement>
So, I can see two issues here - first, there is a difference image, so the results were different, and 2nd, the WallTime & CPUTime are much greater on the failed test. I had been hoping the reason for failure might just be timing-related, and maybe it is, but I had a quick look and I can't see anything in the TestMultipleRenderers.cxx which stipulates any strict time limit for the test, so I don't think that's it.
Perhaps the reason these tests are failing is obvious to someone with a bit more knowledge of the testing process? I've got a feeling the problem is probably something pretty obvious that I'm just overlooking.
Or if not, does anyone have any suggestions re how I could get some more info to debug the issue?
Ta for any advice!
ps if it helps, the full results of a build with all these inexplicably failing tests can be found at: https://homunculoid.com/CDash/viewTest.php?buildid=29