FetchContent and find_package() integration
This is a proposal for adding a degree of integration between the FetchContent
module and the find_package()
command. A proof-of-concept merge request is available in !5688 (merged).
Motivation
FetchContent
is primarily about bringing in external content and adding it to the build. It has strong features for dependency management, but it lacks support for pre-built binaries. find_package()
is a complement to that, being primarily focused on bringing pre-built binaries into the build but with no support for building from source. A natural desire is to be able to use pre-built binaries if available and fall back to building from source if not available. Basically try find_package()
and if that fails, use FetchContent
to fetch and build.
There is also a scenario which is somewhat of the reverse of the above. Continuous integration builds may wish to ensure that they don't pick up random packages from the build machine. They may explicitly want to use dependencies built from source via FetchContent
instead of using find_package()
. When the dependency hierarchy is deep and involves third party packages, it can be difficult to achieve this without hacking away at those dependency projects which have their own find_package()
calls or resorting to a traditional super build structure based on ExternalProject
(or worse, overriding the find_package()
command with a different implementation, which is unsupported behavior).
Another common situation based on anecdotal evidence and personal discussions is that Linux distributions and third party package managers tend to prefer find_package()
. Their workflows typically handle organising dependencies themselves and they manage the build steps such that binaries should be available (or they are built on demand by the tool itself). Feedback from some of these groups has been that where projects use FetchContent
, these tools or workflows may need to hack the project to not do that because it doesn't fit with their find_package()
-based workflow. A follow-up issue will address this particular topic more directly, but I'm mentioning it here for added context.
Proposal
- Give
FetchContent_Declare()
the ability to specify arguments for afind_package()
call.FetchContent_MakeAvailable()
can then attempt a call tofind_package()
and if that succeeds, use that instead of the usual population. This implements the prefer-pre-built-binaries scenario. - Provide a cache variable (proposed name
FETCHCONTENT_TRY_FIND_PROJECT_MODE
) which can be used to control the default behavior ofFetchContent_Declare()
calls that don't specifyfind_package()
arguments. Supported values would beALWAYS
,NEVER
andOPT_IN
(names are up for discussion). Default would beOPT_IN
and an empty value also meansOPT_IN
to preserve backward compatibility. This provides a potentially easier path for adoption where dependencies meet certain criteria (see discussion below). - Give
FetchContent_Declare()
the ability to specify thatfind_package()
calls for a dependency should useFetchContent_MakeAvailable()
instead. This implements the prefer-build-from-source scenario.
Discussion
For at least a few years now, a standard pattern that has been emerging is that projects should aim to support being brought into a build via find_package()
or by adding its source directory to a build via add_subdirectory()
. The same set of targets should be available with either method and there are established patterns for doing that (exporting with a namespace, using alias targets to reproduce the same exported targets for from-source builds, etc.). For dependencies that follow this philosophy, the above proposal should work quite cleanly. Since commands have global visibility, even functions and macros added via either find_package()
or add_subdirectory()
should present no significant problems.
Of course, the above proposed behavior only works if projects are not closely coupled to the particulars of find_package()
or FetchContent
. For example, packages found via find modules will frequently define variables rather than imported targets (blah_INCLUDE_DIRS
, blah_LIBRARIES
, etc.). Projects using FetchContent
may have logic that makes use of the depname_SOURCE_DIR
variable defined by FetchContent_MakeAvailable()
and friends.
This proposal acknowledges that it won't be possible to use the proposed features in all cases. Only if the dependency projects are known to not rely on variables provided by find_package()
or FetchContent
(or could be made to be so) would the user be able to enable these features. That still seems like a large potential pool and as more projects adopt the more modern target-based approach, the situation should organically improve over time.
The FETCHCONTENT_TRY_FIND_PROJECT_MODE
variable is meant as a convenience. One can incrementally adopt these features without it, but it may reduce the work needed and potentially allow existing projects to adopt the new features with only modest or no changes. The NEVER
mode also provides a way to switch off the "try find_package() first" feature if projects are using dependencies that started introducing these features but they don't work in the current user's broader project. It's a way to restore behavior to what we have now in this regard. There currently isn't a global switch for the reverse direction (redirecting find_package()
to FetchContent_MakeAvailable()
), this still has to be opt-in on a dependency-by-dependency basis. It isn't clear whether blocking a FetchContent
from overriding find_package()
is any safer than allowing that override - either way you have to check what the project is doing. I'm happy to consider adding such a global control if feedback and discussion concludes that it is desirable.
Implementation Details
When a dependency is populated by FetchContent
, we need a way to convince find_package()
to use that populated content. The FetchContent
module can write out config and config-version files to satisfy find_package()
, with the assumption that the dependency produces the same imported targets when built from source as it does when brought in via find_package()
. As an extra hook for workarounds, the config file that FetchContent
generates could look for a supplemental file that projects can supply and that supplemental file could also be loaded. It could define any required variables, package version details, etc. such that it behaves just like a regular find_package()
. I wouldn't expect this facility to be needed frequently when working with projects using modern practices, but it could be handy for dealing with older dependencies.
In putting together the proof-of-concept, I provide a dedicated area in the build directory that is regenerated from scratch every time CMake is run (currently in CMakeFiles/pkgRedirects
, which CMake clears at the start of every run). As FetchContent
populates a dependency, it has the opportunity to write custom config files for that dependency into the pkgRedirects
directory. The find_package()
implementation has been updated to look there first for config mode so that it will take precedence. Projects can write the extra supplemental files into that same area as part of their normal processing, they would just need to do it before anything tried to call find_package()
on that dependency. If required, even the config files can be provided by the project or even the dependency itself, since the location of this redirects dir is available to it. See the associated MR for details of how this works. Wiping this directory each time is important for ensuring robust behavior as a build directory is used over time. Projects might not always write a supplemental file (e.g. developer checking out different branches) so you can't safely leave behind things for a subsequent run.
One current drawback I've encountered is that we don't know the package version if the dependency is brought directly into the build via FetchContent
instead of find_package()
. The supplemental file could provide it, but that's something we might be able to improve if we added a new keyword to FetchContent_Declare()
for specifying the version directly there. That could potentially be used instead of things like GIT_TAG
, but it would require implementing some logic for querying whether well-known tag formats existed matching the provided version number. That feels like more than I'd want to bite off in a first implementation though. It could be added as an improvement later.
Related Issues
Aspects of this have appeared in discussions in the following issues (there are probably more):
- #17735 (probably the main one, a lot of the ideas here were originally discussed/proposed in this issue)
- #20852 (closed) (will essentially be closed as superceded by this one)