Random crashes when linking with msvc 2015
Hello,
We face random crashes in cmake. The crashes are random and not reproducible. The issue mainly happens in CI due to large number of builds we perform daily.
Build environment
- Windows server 2016
- Visual studio 2015
- qt 5.9.2
- Cmake v3.17.4
- Ninja 1.10
Details
The crash never happened during compiling c++. It only happens during linking, moc generation or header generation using uic.
Linking crash example
FAILED: bin/FormatPrimaryPeptidesHashTest.exe
cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=common\tests\auto\CMakeFiles\FormatPrimaryPeptidesHashTest.dir --rc=C:\PROGRA~2\WI3CF2~1\8.1\bin\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\8.1\bin\x64\mt.exe --manifests -- C:\PROGRA~2\MICROS~1.0\VC\bin\amd64\link.exe /nologo @CMakeFiles\FormatPrimaryPeptidesHashTest.rsp /out:bin\FormatPrimaryPeptidesHashTest.exe /implib:lib\FormatPrimaryPeptidesHashTest.lib /pdb:bin\FormatPrimaryPeptidesHashTest.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console /MANIFEST:NO && cmd.exe /C "cd /D C:\pmi\pmi_qt_apps_libs\build\common\tests\auto && "C:\Program Files\CMake\bin\cmake.exe" -DPMI_ADD_MANIFEST_APP="FormatPrimaryPeptidesHashTest.exe" -DPMI_OUTPUT_MANIFEST_FILE="C:/pmi/pmi_qt_apps_libs/build/bin/FormatPrimaryPeptidesHashTest.exe.manifest" -DPMI_MANIFEST_TEMPLATE="C:/.conan/ffa1c0/1/lib/cmake/pmi_qt_common/PMIApp.exe.manifest.in" -DPMI_REL_MANIFEST_TEMPLATE="../../../.conan/ffa1c0/1/lib/cmake/pmi_qt_common/PMIApp.exe.manifest.in" -P C:/.conan/38b1c7/1/cmake/modules/PMIAddManifest.cmake""
ninja: build stopped: subcommand failed.
Moc crash example
FAILED: Tools/enterprise/tests/manual/CMakeFiles/Enterprise2020UploaderManualTest_autogen Tools/enterprise/tests/manual/Enterprise2020UploaderManualTest_autogen/mocs_compilation.cpp
cmd.exe /C "cd /D C:\pmi\pmi_qt_apps_libs\build\Tools\enterprise\tests\manual && "C:\Program Files\CMake\bin\cmake.exe" -E cmake_autogen C:/pmi/pmi_qt_apps_libs/build/Tools/enterprise/tests/manual/CMakeFiles/Enterprise2020UploaderManualTest_autogen.dir/AutogenInfo.json RelWithDebInfo"
ninja: build stopped: subcommand failed.
UIC crash example
FAILED: common_pmap/ui_BottomUpDialog.h
cmd.exe /C "cd /D C:\pmi\pmi_qt_apps_libs\build\common_pmap && C:\.conan\198c36\1\bin\uic.exe -o C:/pmi/pmi_qt_apps_libs/build/common_pmap/ui_BottomUpDialog.h C:/pmi/pmi_qt_apps_libs/source/common_pmap/src/ui/BottomUpDialog.ui"
ninja: build stopped: subcommand failed.
Attempts to fix it
Since there is no error message, most of the things we tried were based on guesses.
- We tried upgrading to the latest cmake version (3.18) but the crash become more frequent (About 5-10% of the build failed), so we started downgrading cmake, the issue kept happening with the same frequence with all the versions down to 3.16.8
With the 3.16.8 the crash happens for less than 1% of the builds (but that still too much due to the number of the builds). We couldn't downgrade furthermore as we use some features that are only available since 3.16.8
- We waited for the crash to happen, logged in to the CI machine and executed the exact command that failed manually but it succeeded. We tried this couple of times but never were able to reproduce the crash.
How to proceed from here ?
- Is this maybe a known issue ?
- Do you recommend any way to debug this issue ?
Thanks. Walid