Assuming we should not use
llvm-mt
yet, please try the change in !7054 (merged).
I did a fresh build of clang with !7054 (merged) and I can confirm it clears clang unit test failures we saw previously. Shall we land this patch?
Thanks for the comments.
For reference,
llvm-mt
has been preferred overmt
since !6164 (merged).
We didn't use latest cmake on our builders previously. The issue shows up once we pin our cmake to the latest release version. We currently mitigate this issue by manually setting CMAKE_MT
to mt.exe
.
Assuming we should not use
llvm-mt
yet, please try the change in !7054 (merged).
I will do a test as soon as possible.
One can see most of the options we pass to
mt
in the code here. One of the flag is definitely not supported yet. If you runllvm-mt.exe /?
it will print:
/notify_update Not supported
It is related to the special return code when manifest file is updated. Could that potentially be the cause of the potential data race?
I am debugging a weird llvm test failures in our builder which popped up a few month ago, which is caused by the fact that an executable generated during llvm's cmake configuration step could not be executed, so the tests won't get the correct error string on Windows. An example test failure can seen https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-windows-x64/b8832262407257821617/overview after cmake switched to llvm-mt by default in patch https://github.com/Kitware/CMake/commit/b12aec6c8daa3e087e6d0fa0441f59622251eb46. At first I thought it was a llvm-mt.exe bug. But after I managed to generate a minimal reproducer, it seems to be an issue with cmake's vs_link_exe feature.
How to reproduce the issue:
clang-cl.exe
, clang++.exe
to clang.exe
and lld-link.exe
to lld.exe
as some ZIP program does not process symlinks on windows correctly)cmake-3.23.0-rc2-windows-x86_64.zip
)llvm-mt-reproduce.zip
from this ticket and extract it.C:\src\clang\windows-amd64\bin
and cmake.exe is placed under C:\src\cmake-3.23.0-rc2-windows-x86_64\bin
, open x64 Native Tools Command Prompt for VS
from the start menu.C:\src\llvm-mt-reproduce>run-with-cmake.bat
...
C:\src\llvm-mt-reproduce>out\getErrc.exe
Access is denied.
C:\src\llvm-mt-reproduce>
The getErrc.exe
is a program generated during llvm's cmake configuration step to extract the correct error message string for llvm's unit tests. When this step fails, all error message related unit test will fail, causing the test failures we saw on our builders. The content of the batch file used in the reproducer package is listed as follow:
SETLOCAL
set CLANG_CL_PATH=C:\src\clang\windows-amd64\bin\clang-cl.exe
set CMAKE_PATH=C:\src\cmake-3.23.0-rc2-windows-x86_64\bin\cmake.exe
set MT_PATH=C:\src\clang\windows-amd64\bin\llvm-mt.exe
::set MT_PATH="C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64\mt.exe"
set RC_PATH="C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64\rc.exe"
::set RC_PATH=C:\src\clang\windows-amd64\bin\llvm-rc.exe
set LINKER_PATH=C:\src\clang\windows-amd64\bin\lld-link.exe
%CLANG_CL_PATH% /nologo -TP /DWIN32 /D_WINDOWS /Zc:inline /Zc:__cplusplus /Zc:strictStrings /Oi /Zc:rvalueCast /Brepro /bigobj /W4 -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation /Gw -no-canonical-prefixes /MDd /Zi /Ob0 /Od /RTC1 -std:c++14 /showIncludes /Foout\getErrc.cpp.obj /Fdout\ -c -- .\getErrc.cpp
%CMAKE_PATH% -E vs_link_exe --intdir=out --rc=%RC_PATH% --mt=%MT_PATH% --manifests -- %LINKER_PATH% /nologo out\getErrc.cpp.obj /out:out\getErrc.exe /implib:out\getErrc.lib /pdb:out\getErrc.pdb /version:0.0 /machine:x64 /STACK:10000000 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib
You can change the paths in run-with-cmake.bat
file if your paths does not match mine.
On my machine, the getErrc.exe
generated from cmake's vs_link step cannot be executed and will always result in "this app cannot run on your PC" error and "Access is denied" will be printed to the cmd. How ever, if you run run-with-cmake.bat
again (to be precise, to run the cmake -E vs_link_exe ...
from the batch file), the out\getErrc.exe
generated by the 2nd run will work without any issues:
C:\src\llvm-mt-reproduce> C:\src\cmake-3.23.0-rc2-windows-x86_64\bin\cmake.exe -E vs_link_exe --intdir=out --rc="C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64\rc.exe" --mt=C:\src\clang\windows-amd64\bin\llvm-mt.exe --manifests -- C:\src\clang\windows-amd64\bin\lld-link.exe /nologo out\getErrc.cpp.obj /out:out\getErrc.exe /implib:out\getErrc.lib /pdb:out\getErrc.pdb /version:0.0 /machine:x64 /STACK:10000000 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib
C:\src\llvm-mt-reproduce>out\getErrc.exe
no such file or directory;is a directory;invalid argument;permission denied
C:\src\llvm-mt-reproduce>
So it make me suspect that there might be an data race issue here. When the cmake -E vs_link_exe ...
was invoked the first time, an incomplete manifest file that was still being written might have been present to the linker and embedded to the final getErrc.exe, causing the error messages we saw. When running the command twice, the manifest file were already correctly generated, so this time the executable has the correct manifest file embedded. If this is the case, I think this is a bug on cmake side and should be fixed.
A few other observations:
--trace
flag to the cmake -E vs_link_exe ...
(so final command line looks like ) to get the command traces, the original successful cmake invocation will result in error with messages:Running with trace output on.
CMake Warning:
Ignoring extra path from command line:
"advapi32.lib"
CMake Error: Unknown argument --
CMake Error: Run 'cmake --help' for all supported options.
Is it a cmake flag parser bug? I have to use Windows audit mode to record the command line flags in order to debug this failure.
/manifest
and /out
flags are implemented, others are not) compared to the original mt.exe
provided by Windows MSVC. Is it too early for cmake to set llvm-mt by default on Windows? Can we switch it back to the mt.exe
?