FetchContent and find_package() integration
This is a proposal for adding a degree of integration between the
FetchContent module and the
find_package() command. A proof-of-concept merge request is available in !5688 (closed).
FetchContent is primarily about bringing in external content and adding it to the build. It has strong features for dependency management, but it lacks support for pre-built binaries.
find_package() is a complement to that, being primarily focused on bringing pre-built binaries into the build but with no support for building from source. A natural desire is to be able to use pre-built binaries if available and fall back to building from source if not available. Basically try
find_package() and if that fails, use
FetchContent to fetch and build.
There is also a scenario which is somewhat of the reverse of the above. Continuous integration builds may wish to ensure that they don't pick up random packages from the build machine. They may explicitly want to use dependencies built from source via
FetchContent instead of using
find_package(). When the dependency hierarchy is deep and involves third party packages, it can be difficult to achieve this without hacking away at those dependency projects which have their own
find_package() calls or resorting to a traditional super build structure based on
ExternalProject (or worse, overriding the
find_package() command with a different implementation, which is unsupported behavior).
Another common situation based on anecdotal evidence and personal discussions is that Linux distributions and third party package managers tend to prefer
find_package(). Their workflows typically handle organising dependencies themselves and they manage the build steps such that binaries should be available (or they are built on demand by the tool itself). Feedback from some of these groups has been that where projects use
FetchContent, these tools or workflows may need to hack the project to not do that because it doesn't fit with their
find_package()-based workflow. A follow-up issue will address this particular topic more directly, but I'm mentioning it here for added context.
FetchContent_Declare()the ability to specify arguments for a
FetchContent_MakeAvailable()can then attempt a call to
find_package()and if that succeeds, use that instead of the usual population. This implements the prefer-pre-built-binaries scenario.
- Provide a cache variable (proposed name
FETCHCONTENT_TRY_FIND_PROJECT_MODE) which can be used to control the default behavior of
FetchContent_Declare()calls that don't specify
find_package()arguments. Supported values would be
OPT_IN(names are up for discussion). Default would be
OPT_INand an empty value also means
OPT_INto preserve backward compatibility. This provides a potentially easier path for adoption where dependencies meet certain criteria (see discussion below).
FetchContent_Declare()the ability to specify that
find_package()calls for a dependency should use
FetchContent_MakeAvailable()instead. This implements the prefer-build-from-source scenario.
For at least a few years now, a standard pattern that has been emerging is that projects should aim to support being brought into a build via
find_package() or by adding its source directory to a build via
add_subdirectory(). The same set of targets should be available with either method and there are established patterns for doing that (exporting with a namespace, using alias targets to reproduce the same exported targets for from-source builds, etc.). For dependencies that follow this philosophy, the above proposal should work quite cleanly. Since commands have global visibility, even functions and macros added via either
add_subdirectory() should present no significant problems.
Of course, the above proposed behavior only works if projects are not closely coupled to the particulars of
FetchContent. For example, packages found via find modules will frequently define variables rather than imported targets (
blah_LIBRARIES, etc.). Projects using
FetchContent may have logic that makes use of the
depname_SOURCE_DIR variable defined by
FetchContent_MakeAvailable() and friends.
This proposal acknowledges that it won't be possible to use the proposed features in all cases. Only if the dependency projects are known to not rely on variables provided by
FetchContent (or could be made to be so) would the user be able to enable these features. That still seems like a large potential pool and as more projects adopt the more modern target-based approach, the situation should organically improve over time.
FETCHCONTENT_TRY_FIND_PROJECT_MODE variable is meant as a convenience. One can incrementally adopt these features without it, but it may reduce the work needed and potentially allow existing projects to adopt the new features with only modest or no changes. The
NEVER mode also provides a way to switch off the "try find_package() first" feature if projects are using dependencies that started introducing these features but they don't work in the current user's broader project. It's a way to restore behavior to what we have now in this regard. There currently isn't a global switch for the reverse direction (redirecting
FetchContent_MakeAvailable()), this still has to be opt-in on a dependency-by-dependency basis. It isn't clear whether blocking a
FetchContent from overriding
find_package() is any safer than allowing that override - either way you have to check what the project is doing. I'm happy to consider adding such a global control if feedback and discussion concludes that it is desirable.
When a dependency is populated by
FetchContent, we need a way to convince
find_package() to use that populated content. The
FetchContent module can write out config and config-version files to satisfy
find_package(), with the assumption that the dependency produces the same imported targets when built from source as it does when brought in via
find_package(). As an extra hook for workarounds, the config file that
FetchContent generates could look for a supplemental file that projects can supply and that supplemental file could also be loaded. It could define any required variables, package version details, etc. such that it behaves just like a regular
find_package(). I wouldn't expect this facility to be needed frequently when working with projects using modern practices, but it could be handy for dealing with older dependencies.
In putting together the proof-of-concept, I provide a dedicated area in the build directory that is regenerated from scratch every time CMake is run (currently in
CMakeFiles/pkgRedirects, which CMake clears at the start of every run). As
FetchContent populates a dependency, it has the opportunity to write custom config files for that dependency into the
pkgRedirects directory. The
find_package() implementation has been updated to look there first for config mode so that it will take precedence. Projects can write the extra supplemental files into that same area as part of their normal processing, they would just need to do it before anything tried to call
find_package() on that dependency. If required, even the config files can be provided by the project or even the dependency itself, since the location of this redirects dir is available to it. See the associated MR for details of how this works. Wiping this directory each time is important for ensuring robust behavior as a build directory is used over time. Projects might not always write a supplemental file (e.g. developer checking out different branches) so you can't safely leave behind things for a subsequent run.
One current drawback I've encountered is that we don't know the package version if the dependency is brought directly into the build via
FetchContent instead of
find_package(). The supplemental file could provide it, but that's something we might be able to improve if we added a new keyword to
FetchContent_Declare() for specifying the version directly there. That could potentially be used instead of things like
GIT_TAG, but it would require implementing some logic for querying whether well-known tag formats existed matching the provided version number. That feels like more than I'd want to bite off in a first implementation though. It could be added as an improvement later.
Aspects of this have appeared in discussions in the following issues (there are probably more):