ExternalProject: add ability to move dependency tracking from configure time to build time
ExternalProject is very useful when using CMake as a superbuild system, allowing the delegation of building to the build systems of a variety of types, including autotools, CMake, etc.. When using ExternalProject one must specify download method and using this information it can set up some dependencies on given tarballs, git checkout, or any number of different options. This in effect means that you need to declare all of the dependencies up front and must regenerate the CMake rules if they change.
When dealing with many, many ExternalProjects (e.g. 400+), the generation process for the build rules can take a significant amount of time, due to its single threaded nature, easily up to or exceeding a minute. This makes iterative development of sub projects more painful than it could be.
With ExternalProjects, the super CMake project may not be the best place to determine what the sub project's build system need to do and in many cases, the super project build rules will be unaffected if a sub project changes, i.e. updating a tarball without any subsequent change in build rules.
Therefore if the dependency tracking can be moved to build time, it would avoid the need to regenerate build rules, which would end up being the same, thus saving compute time by avoiding this step and making iterative development easier.
This was not possible with vanilla ninja until recently but the recently added feature of dyndeps would allow this, as long as there was the ability to specify a command for ninja to discover the dependencies at build time.
An example sketch might be to have something like this (assuming the addition of a find_files
command to cmake
which would be a platform independent version of find <dir> -type f
):
ExternalProject_Add(foobar
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/foobar
DYNDEPS_CONFIGURE_COMMAND cmake -E find_files <SOURCE_DIR>
)
This example would add all files in that external project's folder as dependencies to the configure step. Obviously the user has to know what they're doing as it would be possible to do something to get non deterministic behaviour and as such would be an advanced feature with a caveat emptor guarantee.
At present I'm using a bespoke temporary engineering solution in a forked ninja to achieve this behaviour and it has shown to us to make the experience of CMake as super build system much better by avoiding the slow configure/generation steps of CMake altogether.