unnecessary dependencies added to files generated through add_custom_command
Hi,
I'm not sure if this is a "bug", me not understanding how add_custom_command
works or something else, but I have noticed in my build system a scenario that doesn't work quite as I expected. Following is an explanation of my problem, and at the end of the issue is a small repro.
I have a set of common libraries (e.g. just a add_library
with some static files, no custom commands or anything fancy)
Then I have a library whose files are generated by a custom command. The generation itself depends on absolutely nothing except their input files (it's basically a Python script that reads some json file and generates a .c + .h files containing some code)
And this library depends on some of the "common libraries" because the generated code uses some of their functionality.
Now, in this scenario, the custom command should have zero dependencies on the common libraries. Only the actual build and link steps that compile and link those files should. But using Ninja (and I also tested with NMake Makefiles) I noticed that all the libraries that are added through target_link_libraries
to the generated lib also appear as dependencies of the custom commands that create the files for that lib.
And this (at least in my case) is wrong: the result is that the build has to wait for all the common libraries before starting to generate the files of the generated lib... In my case I have 3 or 4 common libs that takes quite a while to run. And I have a lot of generated files, whose generation also takes quite a while. So I end up with only a few cores of my machine being used until all the common libs are built, and only then the generation of the files starts, while it could have started in parallel with building/linking the common libs.
Anyway, this might sound trivial, but in our "real life" build system, it makes things a lot slower than it could be.
Now, the repro:
cmake_minimum_required (VERSION 3.7)
project (custom_cmd_deps)
# dummy common library, for example's sake (normally it's a static file)
file (WRITE ${CMAKE_BINARY_DIR}/utils.cc "int bar() { return 3; }")
add_library (utils STATIC ${CMAKE_BINARY_DIR}/utils.cc)
# now I have a library whose files are generated by a custom command. Generating those files in itself
# doesn't depend on anything. It's just a script that reads some data and generate code accordingly.
# Of course in this example it's completely useless, in "real life" it generates a header with forward
# decls, etc.
add_custom_command (
OUTPUT ${CMAKE_BINARY_DIR}/lib.c
COMMAND echo float foo\(\) { return 3.1416f\; } > ${CMAKE_BINARY_DIR}/lib.c
COMMENT "Generating lib.c"
)
# this is the library which compiles the file output by the previous custom command
add_library (lib SHARED ${CMAKE_BINARY_DIR}/lib.c)
# now, in my real scenario, the code generated by the custom command uses some functionalities from `utils`
# The generation itself (the custom command) doesn't, but the compilation of the .c and the link does.
# And the problem is that now, the custom command now waits for `utils` to be fully built before running...
target_link_libraries (lib PRIVATE utils)
If you use the Ninja
generator, you'll notice the custom command deps:
build lib.c: CUSTOM_COMMAND || utils.lib
The result is that in this example, the build occurs on only 1 core. utils
is build then linked, then lib.c
is generated, and lib
is then built and linked.
If I edit the Ninja build file and remove the || utils.lib
part, then the generation of lib.c
is correctly parallelized with building utils
. And if I comment out the last line of the example, the || utils.lib
part goes away. But then in my real scenario, lib
would not build.
So! Sorry for the long explanation, and now to the questions:
- why this dependency? I can't imagine a scenario where it make sense, but I'm not omniscient, so I'm curious to know if there is such a scenario.
- is there a workaround?
- would it be possible to add a flag to
add_custom_command
to make it not add the dependencies of the target that uses its output?
Regards, Damien.