Merge topic 'fetchcontent-reduce-boilerplate'

a94355c7 FetchContent: Add new command FetchContent_MakeAvailable()
611d5274 Sphinx: Add limited support for nested variables in docs
1a07e1b4

 FetchContent: Trivial doc corrections
Acked-by: Kitware Robot <kwrobot@kitware.com>
Merge-request: !2830

Help/release/dev/fetchcontent-reduce-boilerplate.rst

+fetchcontent-reduce-boilerplate
+-------------------------------
+
+* The FetchContent module gained a new :command:`FetchContent_MakeAvailable`
+  command.  This new command accepts a list of dependency names, which it then
+  iterates over, populating and adding each one to the main build using the
+  canonical pattern.  This significantly reduces the amount of boilerplate
+  needed in the project.

Modules/FetchContent.cmake

@@ -20,9 +20,12 @@ configure step to use the content in commands like :command:`add_subdirectory`,
 :command:`include` or :command:`file` operations.
 
 Content population details would normally be defined separately from the
-command that performs the actual population.  Projects should also
-check whether the content has already been populated somewhere else in the
-project hierarchy.  Typical usage would look something like this:
+command that performs the actual population.  This separation ensures that
+all of the dependency details are defined before anything may try to use those
+details to populate content.  This is particularly important in more complex
+project hierarchies where dependencies may be shared between multiple projects.
+
+The following shows a typical example of declaring content details:
 
 .. code-block:: cmake
 
@@ -32,21 +35,37 @@ project hierarchy.  Typical usage would look something like this:
     GIT_TAG        release-1.8.0
   )
 
+For most typical cases, populating the content can then be done with a single
+command like so:
+
+.. code-block:: cmake
+
+  FetchContent_MakeAvailable(googletest)
+
+The above command not only populates the content, it also adds it to the main
+build (if possible) so that the main build can use the populated project's
+targets, etc.  In some cases, the main project may need to have more precise
+control over the population or may be required to explicitly define the
+population steps (e.g. if CMake versions earlier than 3.14 need to be
+supported).  The typical pattern of such custom steps looks like this:
+
+.. code-block:: cmake
+
   FetchContent_GetProperties(googletest)
   if(NOT googletest_POPULATED)
     FetchContent_Populate(googletest)
     add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR})
   endif()
 
-When using the above pattern with a hierarchical project arrangement,
-projects at higher levels in the hierarchy are able to define or override
-the population details of content specified anywhere lower in the project
-hierarchy.  The ability to detect whether content has already been
-populated ensures that even if multiple child projects want certain content
-to be available, the first one to populate it wins.  The other child project
-can simply make use of the already available content instead of repeating
-the population for itself.  See the
-:ref:`Examples <fetch-content-examples>` section which demonstrates
+Regardless of which population method is used, when using the
+declare-populate pattern with a hierarchical project arrangement, projects at
+higher levels in the hierarchy are able to override the population details of
+content specified anywhere lower in the project hierarchy.  The ability to
+detect whether content has already been populated ensures that even if
+multiple child projects want certain content to be available, the first one
+to populate it wins.  The other child project can simply make use of the
+already available content instead of repeating the population for itself.
+See the :ref:`Examples <fetch-content-examples>` section which demonstrates
 this scenario.
 
 The ``FetchContent`` module also supports defining and populating
@@ -113,6 +132,38 @@ Declaring Content Details
 Populating The Content
 ^^^^^^^^^^^^^^^^^^^^^^
 
+For most common scenarios, population means making content available to the
+main build according to previously declared details for that dependency.
+There are two main patterns for populating content, one based on calling
+:command:`FetchContent_GetProperties` and
+:command:`FetchContent_Populate` for more precise control and the other on
+calling :command:`FetchContent_MakeAvailable` for a simpler, more automated
+approach.  The former generally follows this canonical pattern:
+
+.. _`fetch-content-canonical-pattern`:
+
+.. code-block:: cmake
+
+  # Check if population has already been performed
+  FetchContent_GetProperties(<name>)
+  string(TOLOWER "<name>" lcName)
+  if(NOT ${lcName}_POPULATED)
+    # Fetch the content using previously declared details
+    FetchContent_Populate(<name>)
+
+    # Set custom variables, policies, etc.
+    # ...
+
+    # Bring the populated content into the build
+    add_subdirectory(${${lcName}_SOURCE_DIR} ${${lcName}_BINARY_DIR})
+  endif()
+
+The above is such a common pattern that, where no custom steps are needed
+between the calls to :command:`FetchContent_Populate` and
+:command:`add_subdirectory`, equivalent logic can be obtained by calling
+:command:`FetchContent_MakeAvailable` instead (and should be preferred where
+it meets the needs of the project).
+
 .. command:: FetchContent_Populate
 
   .. code-block:: cmake
@@ -309,9 +360,6 @@ Populating The Content
   on the command line invoking the script.
 
 
-Retrieve Population Properties
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
 .. command:: FetchContent_GetProperties
 
   When using saved content details, a call to :command:`FetchContent_Populate`
@@ -343,28 +391,65 @@ Retrieve Population Properties
     FetchContent_GetProperties(foobar)
     if(NOT foobar_POPULATED)
       FetchContent_Populate(foobar)
-
-      # Set any custom variables, etc. here, then
-      # populate the content as part of this build
-
-      add_subdirectory(${foobar_SOURCE_DIR} ${foobar_BINARY_DIR})
+      ...
     endif()
 
   The above pattern allows other parts of the overall project hierarchy to
   re-use the same content and ensure that it is only populated once.
 
 
+.. command:: FetchContent_MakeAvailable
+
+  .. code-block:: cmake
+
+    FetchContent_MakeAvailable( <name1> [<name2>...] )
+
+  This command implements the common pattern typically needed for most
+  dependencies.  It iterates over each of the named dependencies in turn
+  and for each one it loosely follows the same
+  :ref:`canonical pattern <fetch-content-canonical-pattern>` as
+  presented at the beginning of this section.  One small difference to
+  that pattern is that it will only call :command:`add_subdirectory` on the
+  populated content if there is a ``CMakeLists.txt`` file in its top level
+  source directory.  This allows the command to be used for dependencies
+  that make downloaded content available at a known location but which do
+  not need or support being added directly to the build.
+
+
 .. _`fetch-content-examples`:
 
 Examples
 ^^^^^^^^
 
-Consider a project hierarchy where ``projA`` is the top level project and it
-depends on projects ``projB`` and ``projC``. Both ``projB`` and ``projC``
-can be built standalone and they also both depend on another project
-``projD``.  For simplicity, this example will assume that all four projects
-are available on a company git server.  The ``CMakeLists.txt`` of each project
-might have sections like the following:
+This first fairly straightforward example ensures that some popular testing
+frameworks are available to the main build:
+
+.. code-block:: cmake
+
+  include(FetchContent)
+  FetchContent_Declare(
+    googletest
+    GIT_REPOSITORY https://github.com/google/googletest.git
+    GIT_TAG        release-1.8.0
+  )
+  FetchContent_Declare(
+    Catch2
+    GIT_REPOSITORY https://github.com/catchorg/Catch2.git
+    GIT_TAG        v2.5.0
+  )
+
+  # After the following call, the CMake targets defined by googletest and
+  # Catch2 will be defined and available to the rest of the build
+  FetchContent_MakeAvailable(googletest Catch2)
+
+
+In more complex project hierarchies, the dependency relationships can be more
+complicated.  Consider a hierarchy where ``projA`` is the top level project and
+it depends directly on projects ``projB`` and ``projC``.  Both ``projB`` and
+``projC`` can be built standalone and they also both depend on another project
+``projD``.  ``projB`` additionally depends on ``projE``.  This example assumes
+that all five projects are available on a company git server.  The
+``CMakeLists.txt`` of each project might have sections like the following:
 
 *projA*:
 
@@ -373,31 +458,27 @@ might have sections like the following:
   include(FetchContent)
   FetchContent_Declare(
     projB
-    GIT_REPOSITORY git@mycompany.com/git/projB.git
+    GIT_REPOSITORY git@mycompany.com:git/projB.git
     GIT_TAG        4a89dc7e24ff212a7b5167bef7ab079d
   )
   FetchContent_Declare(
     projC
-    GIT_REPOSITORY git@mycompany.com/git/projC.git
+    GIT_REPOSITORY git@mycompany.com:git/projC.git
     GIT_TAG        4ad4016bd1d8d5412d135cf8ceea1bb9
   )
   FetchContent_Declare(
     projD
-    GIT_REPOSITORY git@mycompany.com/git/projD.git
+    GIT_REPOSITORY git@mycompany.com:git/projD.git
     GIT_TAG        origin/integrationBranch
   )
+  FetchContent_Declare(
+    projE
+    GIT_REPOSITORY git@mycompany.com:git/projE.git
+    GIT_TAG        origin/release/2.3-rc1
+  )
 
-  FetchContent_GetProperties(projB)
-  if(NOT projb_POPULATED)
-    FetchContent_Populate(projB)
-    add_subdirectory(${projb_SOURCE_DIR} ${projb_BINARY_DIR})
-  endif()
-
-  FetchContent_GetProperties(projC)
-  if(NOT projc_POPULATED)
-    FetchContent_Populate(projC)
-    add_subdirectory(${projc_SOURCE_DIR} ${projc_BINARY_DIR})
-  endif()
+  # Order is important, see notes in the discussion further below
+  FetchContent_MakeAvailable(projD projB projC)
 
 *projB*:
 
@@ -406,16 +487,16 @@ might have sections like the following:
   include(FetchContent)
   FetchContent_Declare(
     projD
-    GIT_REPOSITORY git@mycompany.com/git/projD.git
+    GIT_REPOSITORY git@mycompany.com:git/projD.git
     GIT_TAG        20b415f9034bbd2a2e8216e9a5c9e632
   )
+  FetchContent_Declare(
+    projE
+    GIT_REPOSITORY git@mycompany.com:git/projE.git
+    GIT_TAG        68e20f674a48be38d60e129f600faf7d
+  )
 
-  FetchContent_GetProperties(projD)
-  if(NOT projd_POPULATED)
-    FetchContent_Populate(projD)
-    add_subdirectory(${projd_SOURCE_DIR} ${projd_BINARY_DIR})
-  endif()
-
+  FetchContent_MakeAvailable(projD projE)
 
 *projC*:
 
@@ -424,48 +505,77 @@ might have sections like the following:
   include(FetchContent)
   FetchContent_Declare(
     projD
-    GIT_REPOSITORY git@mycompany.com/git/projD.git
+    GIT_REPOSITORY git@mycompany.com:git/projD.git
     GIT_TAG        7d9a17ad2c962aa13e2fbb8043fb6b8a
   )
 
+  # This particular version of projD requires workarounds
   FetchContent_GetProperties(projD)
   if(NOT projd_POPULATED)
     FetchContent_Populate(projD)
+
+    # Copy an additional/replacement file into the populated source
+    file(COPY someFile.c DESTINATION ${projd_SOURCE_DIR}/src)
+
     add_subdirectory(${projd_SOURCE_DIR} ${projd_BINARY_DIR})
   endif()
 
 A few key points should be noted in the above:
 
 - ``projB`` and ``projC`` define different content details for ``projD``,
-  but ``projA`` also defines a set of content details for ``projD`` and
-  because ``projA`` will define them first, the details from ``projB`` and
+  but ``projA`` also defines a set of content details for ``projD``.
+  Because ``projA`` will define them first, the details from ``projB`` and
   ``projC`` will not be used.  The override details defined by ``projA``
   are not required to match either of those from ``projB`` or ``projC``, but
   it is up to the higher level project to ensure that the details it does
   define still make sense for the child projects.
-- While ``projA`` defined content details for ``projD``, it did not need
-  to explicitly call ``FetchContent_Populate(projD)`` itself.  Instead, it
-  leaves that to a child project to do (in this case it will be ``projB``
-  since it is added to the build ahead of ``projC``).  If ``projA`` needed to
-  customize how the ``projD`` content was brought into the build as well
-  (e.g. define some CMake variables before calling
-  :command:`add_subdirectory` after populating), it would do the call to
-  ``FetchContent_Populate()``, etc. just as it did for the ``projB`` and
-  ``projC`` content.  For higher level projects, it is usually enough to
-  just define the override content details and leave the actual population
-  to the child projects.  This saves repeating the same thing at each level
-  of the project hierarchy unnecessarily.
-- Even though ``projA`` is the top level project in this example, it still
-  checks whether ``projB`` and ``projC`` have already been populated before
-  going ahead to do those populations.  This makes ``projA`` able to be more
-  easily incorporated as a child of some other higher level project in the
-  future if required.  Always protect a call to
-  :command:`FetchContent_Populate` with a check to
-  :command:`FetchContent_GetProperties`, even in what may be considered a top
-  level project at the time.
-
-
-The following example demonstrates how one might download and unpack a
+- In the ``projA`` call to :command:`FetchContent_MakeAvailable`, ``projD``
+  is listed ahead of ``projB`` and ``projC`` to ensure that ``projA`` is in
+  control of how ``projD`` is populated.
+- While ``projA`` defines content details for ``projE``, it does not need
+  to explicitly call ``FetchContent_MakeAvailable(projE)`` or
+  ``FetchContent_Populate(projD)`` itself.  Instead, it leaves that to the
+  child ``projB``.  For higher level projects, it is often enough to just
+  define the override content details and leave the actual population to the
+  child projects.  This saves repeating the same thing at each level of the
+  project hierarchy unnecessarily.
+
+
+Projects don't always need to add the populated content to the build.
+Sometimes the project just wants to make the downloaded content available at
+a predictable location.  The next example ensures that a set of standard
+company toolchain files (and potentially even the toolchain binaries
+themselves) is available early enough to be used for that same build.
+
+.. code-block:: cmake
+
+  cmake_minimum_required(VERSION 3.14)
+
+  include(FetchContent)
+  FetchContent_Declare(
+    mycom_toolchains
+    URL  https://intranet.mycompany.com//toolchains_1.3.2.tar.gz
+  )
+  FetchContent_MakeAvailable(mycom_toolchains)
+
+  project(CrossCompileExample)
+
+The project could be configured to use one of the downloaded toolchains like
+so:
+
+.. code-block:: shell
+
+  cmake -DCMAKE_TOOLCHAIN_FILE=_deps/mycom_toolchains-src/toolchain_arm.cmake /path/to/src
+
+When CMake processes the ``CMakeLists.txt`` file, it will download and unpack
+the tarball into ``_deps/mycompany_toolchains-src`` relative to the build
+directory.  The :variable:`CMAKE_TOOLCHAIN_FILE` variable is not used until
+the :command:`project` command is reached, at which point CMake looks for the
+named toolchain file relative to the build directory.  Because the tarball has
+already been downloaded and unpacked by then, the toolchain file will be in
+place, even the very first time that ``cmake`` is run in the build directory.
+
+Lastly, the following example demonstrates how one might download and unpack a
 firmware tarball using CMake's :manual:`script mode <cmake(1)>`.  The call to
 :command:`FetchContent_Populate` specifies all the content details and the
 unpacked firmware will be placed in a ``firmware`` directory below the
@@ -921,3 +1031,31 @@ function(FetchContent_Populate contentName)
   set(${contentNameLower}_POPULATED  True PARENT_SCOPE)
 
 endfunction()
+
+# Arguments are assumed to be the names of dependencies that have been
+# declared previously and should be populated. It is not an error if
+# any of them have already been populated (they will just be skipped in
+# that case). The command is implemented as a macro so that the variables
+# defined by the FetchContent_GetProperties() and FetchContent_Populate()
+# calls will be available to the caller.
+macro(FetchContent_MakeAvailable)
+
+  foreach(contentName IN ITEMS ${ARGV})
+    string(TOLOWER ${contentName} contentNameLower)
+    FetchContent_GetProperties(${contentName})
+    if(NOT ${contentNameLower}_POPULATED)
+      FetchContent_Populate(${contentName})
+
+      # Only try to call add_subdirectory() if the populated content
+      # can be treated that way. Protecting the call with the check
+      # allows this function to be used for projects that just want
+      # to ensure the content exists, such as to provide content at
+      # a known location.
+      if(EXISTS ${${contentNameLower}_SOURCE_DIR}/CMakeLists.txt)
+        add_subdirectory(${${contentNameLower}_SOURCE_DIR}
+                         ${${contentNameLower}_BINARY_DIR})
+      endif()
+    endif()
+  endforeach()
+
+endmacro()

Tests/RunCMake/FetchContent/MakeAvailable-stdout.txt

+Confirmation project has been added
+.*Confirmation script has been called

Tests/RunCMake/FetchContent/MakeAvailable.cmake

+include(FetchContent)
+
+FetchContent_Declare(
+  WithProject
+  SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/WithProject
+)
+FetchContent_Declare(
+  WithoutProject
+  SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/WithoutProject
+)
+
+# Order is important and will be verified by test output
+FetchContent_MakeAvailable(WithProject WithoutProject)
+
+get_property(addedWith GLOBAL PROPERTY FetchWithProject SET)
+if(NOT addedWith)
+  message(SEND_ERROR "Subdir with CMakeLists.txt not added")
+endif()
+
+include(${withoutproject_SOURCE_DIR}/confirmMessage.cmake)

Tests/RunCMake/FetchContent/MakeAvailableTwice-stdout.txt

+-- Before first[
+ ]+-- Confirmation project has been added[
+ ]+-- Between both[
+ ]+-- After last

Tests/RunCMake/FetchContent/MakeAvailableTwice.cmake

+include(FetchContent)
+
+FetchContent_Declare(
+  WithProject
+  SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/WithProject
+)
+
+message(STATUS "Before first")
+FetchContent_MakeAvailable(WithProject)
+message(STATUS "Between both")
+FetchContent_MakeAvailable(WithProject)
+message(STATUS "After last")

Tests/RunCMake/FetchContent/MakeAvailableUndeclared-result.txt

+1

Tests/RunCMake/FetchContent/MakeAvailableUndeclared-stderr.txt

+No content details recorded for NoDetails

Tests/RunCMake/FetchContent/MakeAvailableUndeclared.cmake

+include(FetchContent)
+
+FetchContent_MakeAvailable(NoDetails)

Tests/RunCMake/FetchContent/RunCMakeTest.cmake

@@ -11,6 +11,9 @@ run_cmake(VarDefinitions)
 run_cmake(GetProperties)
 run_cmake(DirOverrides)
 run_cmake(UsesTerminalOverride)
+run_cmake(MakeAvailable)
+run_cmake(MakeAvailableTwice)
+run_cmake(MakeAvailableUndeclared)
 
 # We need to pass through CMAKE_GENERATOR and CMAKE_MAKE_PROGRAM
 # to ensure the test can run on machines where the build tool

Tests/RunCMake/FetchContent/WithProject/CMakeLists.txt

+cmake_minimum_required(VERSION 3.13)
+project(WithProject LANGUAGES NONE)
+
+set_property(GLOBAL PROPERTY FetchWithProject YES)
+message(STATUS "Confirmation project has been added")

Tests/RunCMake/FetchContent/WithoutProject/confirmMessage.cmake

+message(STATUS "Confirmation script has been called")

Utilities/Sphinx/cmake.py

@@ -21,6 +21,8 @@ from pygments.lexer import bygroups
 # - Unix paths are recognized by '/'; support for Windows paths may be added if needed
 # - (\\.) allows for \-escapes (used in manual/cmake-language.7)
 # - $<..$<..$>..> nested occurence in cmake-buildsystem
+# - Nested variable evaluations are only supported in a limited capacity. Only
+#   one level of nesting is supported and at most one nested variable can be present.
 
 CMakeLexer.tokens["root"] = [
   (r'\b(\w+)([ \t]*)(\()', bygroups(Name.Function, Text, Name.Function), '#push'),     # fctn(
@@ -34,7 +36,8 @@ CMakeLexer.tokens["root"] = [
   (r'[<>]=', Punctuation),                                  # used in FindPkgConfig.cmake
   (r'\$<', Operator, '#push'),                              # $<...>
   (r'<[^<|]+?>(\w*\.\.\.)?', Name.Variable),                # <expr>
-  (r'(\$\w*\{)(.+?)(\})', bygroups(Operator, Name.Tag, Operator)),   # ${..} $ENV{..}
+  (r'(\$\w*\{)([^\}\$]*)?(?:(\$\w*\{)([^\}]+?)(\}))?([^\}]*?)(\})',  # ${..} $ENV{..}, possibly nested
+    bygroups(Operator, Name.Tag, Operator, Name.Tag, Operator, Name.Tag, Operator)),
   (r'([A-Z]+\{)(.+?)(\})', bygroups(Operator, Name.Tag, Operator)),  # DATA{ ...}
   (r'[a-z]+(@|(://))((\\.)|[\w.+-:/\\])+', Name.Attribute),          # URL, git@, ...
   (r'/\w[\w\.\+-/\\]*', Name.Attribute),                    # absolute path