ExternalData.cmake 39 KB
Newer Older
1 2 3 4
#[=======================================================================[.rst:
ExternalData
------------

5 6 7 8
.. only:: html

   .. contents::

9 10
Manage data files stored outside source tree

11 12 13
Introduction
^^^^^^^^^^^^

14 15 16 17 18 19 20
Use this module to unambiguously reference data files stored outside
the source tree and fetch them at build time from arbitrary local and
remote content-addressed locations.  Functions provided by this module
recognize arguments with the syntax ``DATA{<name>}`` as references to
external data, replace them with full paths to local copies of those
data, and create build rules to fetch and update the local copies.

21
For example:
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

.. code-block:: cmake

 include(ExternalData)
 set(ExternalData_URL_TEMPLATES "file:///local/%(algo)/%(hash)"
                                "file:////host/share/%(algo)/%(hash)"
                                "http://data.org/%(algo)/%(hash)")
 ExternalData_Add_Test(MyData
   NAME MyTest
   COMMAND MyExe DATA{MyInput.png}
   )
 ExternalData_Add_Target(MyData)

When test ``MyTest`` runs the ``DATA{MyInput.png}`` argument will be
replaced by the full path to a real instance of the data file
``MyInput.png`` on disk.  If the source tree contains a content link
such as ``MyInput.png.md5`` then the ``MyData`` target creates a real
``MyInput.png`` in the build tree.

41 42
Module Functions
^^^^^^^^^^^^^^^^
43

44
.. command:: ExternalData_Expand_Arguments
45

46 47
  The ``ExternalData_Expand_Arguments`` function evaluates ``DATA{}``
  references in its arguments and constructs a new list of arguments::
48

49 50 51 52 53
    ExternalData_Expand_Arguments(
      <target>   # Name of data management target
      <outVar>   # Output variable
      [args...]  # Input arguments, DATA{} allowed
      )
54

55 56
  It replaces each ``DATA{}`` reference in an argument with the full path of
  a real data file on disk that will exist after the ``<target>`` builds.
57

58
.. command:: ExternalData_Add_Test
59

60 61 62
  The ``ExternalData_Add_Test`` function wraps around the CMake
  :command:`add_test` command but supports ``DATA{}`` references in
  its arguments::
63

64 65 66 67
    ExternalData_Add_Test(
      <target>   # Name of data management target
      ...        # Arguments of add_test(), DATA{} allowed
      )
68

69 70
  It passes its arguments through ``ExternalData_Expand_Arguments`` and then
  invokes the :command:`add_test` command using the results.
71

72
.. command:: ExternalData_Add_Target
73

74 75 76 77 78 79 80 81 82
  The ``ExternalData_Add_Target`` function creates a custom target to
  manage local instances of data files stored externally::

    ExternalData_Add_Target(
      <target>   # Name of data management target
      )

  It creates custom commands in the target as necessary to make data
  files available for each ``DATA{}`` reference previously evaluated by
83 84 85 86 87
  other functions provided by this module.
  Data files may be fetched from one of the URL templates specified in
  the ``ExternalData_URL_TEMPLATES`` variable, or may be found locally
  in one of the paths specified in the ``ExternalData_OBJECT_STORES``
  variable.
88

89 90
Module Variables
^^^^^^^^^^^^^^^^
91

92 93
The following variables configure behavior.  They should be set before
calling any of the functions provided by this module.
94

95
.. variable:: ExternalData_BINARY_ROOT
96

97 98 99 100
  The ``ExternalData_BINARY_ROOT`` variable may be set to the directory to
  hold the real data files named by expanded ``DATA{}`` references.  The
  default is ``CMAKE_BINARY_DIR``.  The directory layout will mirror that of
  content links under ``ExternalData_SOURCE_ROOT``.
101

102 103 104 105 106 107
.. variable:: ExternalData_CUSTOM_SCRIPT_<key>

  Specify a full path to a ``.cmake`` custom fetch script identified by
  ``<key>`` in entries of the ``ExternalData_URL_TEMPLATES`` list.
  See `Custom Fetch Scripts`_.

108
.. variable:: ExternalData_LINK_CONTENT
109

110 111 112 113 114 115 116 117 118
  The ``ExternalData_LINK_CONTENT`` variable may be set to the name of a
  supported hash algorithm to enable automatic conversion of real data
  files referenced by the ``DATA{}`` syntax into content links.  For each
  such ``<file>`` a content link named ``<file><ext>`` is created.  The
  original file is renamed to the form ``.ExternalData_<algo>_<hash>`` to
  stage it for future transmission to one of the locations in the list
  of URL templates (by means outside the scope of this module).  The
  data fetch rule created for the content link will use the staged
  object if it cannot be found using any URL template.
119

120 121 122 123 124 125 126
.. variable:: ExternalData_NO_SYMLINKS

  The real data files named by expanded ``DATA{}`` references may be made
  available under ``ExternalData_BINARY_ROOT`` using symbolic links on
  some platforms.  The ``ExternalData_NO_SYMLINKS`` variable may be set
  to disable use of symbolic links and enable use of copies instead.

127
.. variable:: ExternalData_OBJECT_STORES
128

129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164
  The ``ExternalData_OBJECT_STORES`` variable may be set to a list of local
  directories that store objects using the layout ``<dir>/%(algo)/%(hash)``.
  These directories will be searched first for a needed object.  If the
  object is not available in any store then it will be fetched remotely
  using the URL templates and added to the first local store listed.  If
  no stores are specified the default is a location inside the build
  tree.

.. variable:: ExternalData_SERIES_PARSE
              ExternalData_SERIES_PARSE_PREFIX
              ExternalData_SERIES_PARSE_NUMBER
              ExternalData_SERIES_PARSE_SUFFIX
              ExternalData_SERIES_MATCH

  See `Referencing File Series`_.

.. variable:: ExternalData_SOURCE_ROOT

  The ``ExternalData_SOURCE_ROOT`` variable may be set to the highest source
  directory containing any path named by a ``DATA{}`` reference.  The
  default is ``CMAKE_SOURCE_DIR``.  ``ExternalData_SOURCE_ROOT`` and
  ``CMAKE_SOURCE_DIR`` must refer to directories within a single source
  distribution (e.g.  they come together in one tarball).

.. variable:: ExternalData_TIMEOUT_ABSOLUTE

  The ``ExternalData_TIMEOUT_ABSOLUTE`` variable sets the download
  absolute timeout, in seconds, with a default of ``300`` seconds.
  Set to ``0`` to disable enforcement.

.. variable:: ExternalData_TIMEOUT_INACTIVITY

  The ``ExternalData_TIMEOUT_INACTIVITY`` variable sets the download
  inactivity timeout, in seconds, with a default of ``60`` seconds.
  Set to ``0`` to disable enforcement.

165 166 167 168 169 170 171 172
.. variable:: ExternalData_URL_ALGO_<algo>_<key>

  Specify a custom URL component to be substituted for URL template
  placeholders of the form ``%(algo:<key>)``, where ``<key>`` is a
  valid C identifier, when fetching an object referenced via hash
  algorithm ``<algo>``.  If not defined, the default URL component
  is just ``<algo>`` for any ``<key>``.

173 174 175 176 177 178
.. variable:: ExternalData_URL_TEMPLATES

  The ``ExternalData_URL_TEMPLATES`` may be set to provide a list of
  of URL templates using the placeholders ``%(algo)`` and ``%(hash)``
  in each template.  Data fetch rules try each URL template in order
  by substituting the hash algorithm name for ``%(algo)`` and the hash
179 180 181
  value for ``%(hash)``.  Alternatively one may use ``%(algo:<key>)``
  with ``ExternalData_URL_ALGO_<algo>_<key>`` variables to gain more
  flexibility in remote URLs.
182

183 184 185 186 187 188 189 190 191 192 193 194 195 196
Referencing Files
^^^^^^^^^^^^^^^^^

Referencing Single Files
""""""""""""""""""""""""

The ``DATA{}`` syntax is literal and the ``<name>`` is a full or relative path
within the source tree.  The source tree must contain either a real
data file at ``<name>`` or a "content link" at ``<name><ext>`` containing a
hash of the real file using a hash algorithm corresponding to ``<ext>``.
For example, the argument ``DATA{img.png}`` may be satisfied by either a
real ``img.png`` file in the current source directory or a ``img.png.md5``
file containing its MD5 sum.

197
Referencing File Series
198
"""""""""""""""""""""""
199

200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227
The ``DATA{}`` syntax can be told to fetch a file series using the form
``DATA{<name>,:}``, where the ``:`` is literal.  If the source tree
contains a group of files or content links named like a series then a
reference to one member adds rules to fetch all of them.  Although all
members of a series are fetched, only the file originally named by the
``DATA{}`` argument is substituted for it.  The default configuration
recognizes file series names ending with ``#.ext``, ``_#.ext``, ``.#.ext``,
or ``-#.ext`` where ``#`` is a sequence of decimal digits and ``.ext`` is
any single extension.  Configure it with a regex that parses ``<number>``
and ``<suffix>`` parts from the end of ``<name>``::

 ExternalData_SERIES_PARSE = regex of the form (<number>)(<suffix>)$

For more complicated cases set::

 ExternalData_SERIES_PARSE = regex with at least two () groups
 ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any
 ExternalData_SERIES_PARSE_NUMBER = <number> regex group number
 ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number

Configure series number matching with a regex that matches the
``<number>`` part of series members named ``<prefix><number><suffix>``::

 ExternalData_SERIES_MATCH = regex matching <number> in all series members

Note that the ``<suffix>`` of a series does not include a hash-algorithm
extension.

228
Referencing Associated Files
229
""""""""""""""""""""""""""""
230

231 232 233 234 235 236 237 238 239 240 241 242 243
The ``DATA{}`` syntax can alternatively match files associated with the
named file and contained in the same directory.  Associated files may
be specified by options using the syntax
``DATA{<name>,<opt1>,<opt2>,...}``.  Each option may specify one file by
name or specify a regular expression to match file names using the
syntax ``REGEX:<regex>``.  For example, the arguments::

 DATA{MyData/MyInput.mhd,MyInput.img}                   # File pair
 DATA{MyData/MyFrames00.png,REGEX:MyFrames[0-9]+\\.png} # Series

will pass ``MyInput.mha`` and ``MyFrames00.png`` on the command line but
ensure that the associated files are present next to them.

244
Referencing Directories
245
"""""""""""""""""""""""
246

247 248 249 250 251 252 253
The ``DATA{}`` syntax may reference a directory using a trailing slash and
a list of associated files.  The form ``DATA{<name>/,<opt1>,<opt2>,...}``
adds rules to fetch any files in the directory that match one of the
associated file options.  For example, the argument
``DATA{MyDataDir/,REGEX:.*}`` will pass the full path to a ``MyDataDir``
directory on the command line and ensure that the directory contains
files corresponding to every file or content link in the ``MyDataDir``
254 255
source directory.  In order to match associated files in subdirectories,
specify a ``RECURSE:`` option, e.g. ``DATA{MyDataDir/,RECURSE:,REGEX:.*}``.
256

257 258
Hash Algorithms
^^^^^^^^^^^^^^^
259

260
The following hash algorithms are supported::
261

262 263 264 265 266 267 268 269
 %(algo)     <ext>     Description
 -------     -----     -----------
 MD5         .md5      Message-Digest Algorithm 5, RFC 1321
 SHA1        .sha1     US Secure Hash Algorithm 1, RFC 3174
 SHA224      .sha224   US Secure Hash Algorithms, RFC 4634
 SHA256      .sha256   US Secure Hash Algorithms, RFC 4634
 SHA384      .sha384   US Secure Hash Algorithms, RFC 4634
 SHA512      .sha512   US Secure Hash Algorithms, RFC 4634
270

271 272
Note that the hashes are used only for unique data identification and
download verification.
273

274 275
.. _`ExternalData Custom Fetch Scripts`:

276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312
Custom Fetch Scripts
^^^^^^^^^^^^^^^^^^^^

When a data file must be fetched from one of the URL templates
specified in the ``ExternalData_URL_TEMPLATES`` variable, it is
normally downloaded using the :command:`file(DOWNLOAD)` command.
One may specify usage of a custom fetch script by using a URL
template of the form ``ExternalDataCustomScript://<key>/<loc>``.
The ``<key>`` must be a C identifier, and the ``<loc>`` must
contain the ``%(algo)`` and ``%(hash)`` placeholders.
A variable corresponding to the key, ``ExternalData_CUSTOM_SCRIPT_<key>``,
must be set to the full path to a ``.cmake`` script file.  The script
will be included to perform the actual fetch, and provided with
the following variables:

.. variable:: ExternalData_CUSTOM_LOCATION

  When a custom fetch script is loaded, this variable is set to the
  location part of the URL, which will contain the substituted hash
  algorithm name and content hash value.

.. variable:: ExternalData_CUSTOM_FILE

  When a custom fetch script is loaded, this variable is set to the
  full path to a file in which the script must store the fetched
  content.  The name of the file is unspecified and should not be
  interpreted in any way.

The custom fetch script is expected to store fetched content in the
file or set a variable:

.. variable:: ExternalData_CUSTOM_ERROR

  When a custom fetch script fails to fetch the requested content,
  it must set this variable to a short one-line message describing
  the reason for failure.

313
#]=======================================================================]
Brad King's avatar
Brad King committed
314 315

#=============================================================================
316
# Copyright 2010-2015 Kitware, Inc.
Brad King's avatar
Brad King committed
317 318 319 320 321 322 323 324 325 326 327 328
#
# Distributed under the OSI-approved BSD License (the "License");
# see accompanying file Copyright.txt for details.
#
# This software is distributed WITHOUT ANY WARRANTY; without even the
# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the License for more information.
#=============================================================================
# (To distribute this file outside of CMake, substitute the full
#  License text for the above reference.)

function(ExternalData_add_test target)
329 330
  # Expand all arguments as a single string to preserve escaped semicolons.
  ExternalData_expand_arguments("${target}" testArgs "${ARGN}")
Brad King's avatar
Brad King committed
331 332 333 334
  add_test(${testArgs})
endfunction()

function(ExternalData_add_target target)
335 336 337
  if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES)
    message(FATAL_ERROR
      "Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!")
Brad King's avatar
Brad King committed
338 339 340 341
  endif()
  if(NOT ExternalData_OBJECT_STORES)
    set(ExternalData_OBJECT_STORES ${CMAKE_BINARY_DIR}/ExternalData/Objects)
  endif()
342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369
  set(_ExternalData_CONFIG_CODE "")

  # Store custom script configuration.
  foreach(url_template IN LISTS ExternalData_URL_TEMPLATES)
    if("${url_template}" MATCHES "^ExternalDataCustomScript://([^/]*)/(.*)$")
      set(key "${CMAKE_MATCH_1}")
      if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$")
        if(ExternalData_CUSTOM_SCRIPT_${key})
          if(IS_ABSOLUTE "${ExternalData_CUSTOM_SCRIPT_${key}}")
            string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n"
              "set(ExternalData_CUSTOM_SCRIPT_${key} \"${ExternalData_CUSTOM_SCRIPT_${key}}\")")
          else()
            message(FATAL_ERROR
              "No ExternalData_CUSTOM_SCRIPT_${key} is not set to a full path:\n"
              " ${ExternalData_CUSTOM_SCRIPT_${key}}")
          endif()
        else()
          message(FATAL_ERROR
            "No ExternalData_CUSTOM_SCRIPT_${key} is set for URL template:\n"
            " ${url_template}")
        endif()
      else()
        message(FATAL_ERROR
          "Bad ExternalDataCustomScript key '${key}' in URL template:\n"
          " ${url_template}\n"
          "The key must be a valid C identifier.")
      endif()
    endif()
370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388

    # Store custom algorithm name to URL component maps.
    if("${url_template}" MATCHES "%\\(algo:([^)]*)\\)")
      set(key "${CMAKE_MATCH_1}")
      if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$")
        string(REPLACE "|" ";" _algos "${_ExternalData_REGEX_ALGO}")
        foreach(algo ${_algos})
          if(DEFINED ExternalData_URL_ALGO_${algo}_${key})
            string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n"
              "set(ExternalData_URL_ALGO_${algo}_${key} \"${ExternalData_URL_ALGO_${algo}_${key}}\")")
          endif()
        endforeach()
      else()
        message(FATAL_ERROR
          "Bad %(algo:${key}) in URL template:\n"
          " ${url_template}\n"
          "The transform name must be a valid C identifier.")
      endif()
    endif()
389 390 391
  endforeach()

  # Store configuration for use by build-time script.
Brad King's avatar
Brad King committed
392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415
  set(config ${CMAKE_CURRENT_BINARY_DIR}/${target}_config.cmake)
  configure_file(${_ExternalData_SELF_DIR}/ExternalData_config.cmake.in ${config} @ONLY)

  set(files "")

  # Set "_ExternalData_FILE_${file}" for each output file to avoid duplicate
  # rules.  Use local data first to prefer real files over content links.

  # Custom commands to copy or link local data.
  get_property(data_local GLOBAL PROPERTY _ExternalData_${target}_LOCAL)
  foreach(entry IN LISTS data_local)
    string(REPLACE "|" ";" tuple "${entry}")
    list(GET tuple 0 file)
    list(GET tuple 1 name)
    if(NOT DEFINED "_ExternalData_FILE_${file}")
      set("_ExternalData_FILE_${file}" 1)
      add_custom_command(
        COMMENT "Generating ${file}"
        OUTPUT "${file}"
        COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR}
                                 -Dfile=${file} -Dname=${name}
                                 -DExternalData_ACTION=local
                                 -DExternalData_CONFIG=${config}
                                 -P ${_ExternalData_SELF}
416
        MAIN_DEPENDENCY "${name}"
Brad King's avatar
Brad King committed
417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446
        )
      list(APPEND files "${file}")
    endif()
  endforeach()

  # Custom commands to fetch remote data.
  get_property(data_fetch GLOBAL PROPERTY _ExternalData_${target}_FETCH)
  foreach(entry IN LISTS data_fetch)
    string(REPLACE "|" ";" tuple "${entry}")
    list(GET tuple 0 file)
    list(GET tuple 1 name)
    list(GET tuple 2 ext)
    set(stamp "${ext}-stamp")
    if(NOT DEFINED "_ExternalData_FILE_${file}")
      set("_ExternalData_FILE_${file}" 1)
      add_custom_command(
        # Users care about the data file, so hide the hash/timestamp file.
        COMMENT "Generating ${file}"
        # The hash/timestamp file is the output from the build perspective.
        # List the real file as a second output in case it is a broken link.
        # The files must be listed in this order so CMake can hide from the
        # make tool that a symlink target may not be newer than the input.
        OUTPUT "${file}${stamp}" "${file}"
        # Run the data fetch/update script.
        COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR}
                                 -Dfile=${file} -Dname=${name} -Dext=${ext}
                                 -DExternalData_ACTION=fetch
                                 -DExternalData_CONFIG=${config}
                                 -P ${_ExternalData_SELF}
        # Update whenever the object hash changes.
447
        MAIN_DEPENDENCY "${name}${ext}"
Brad King's avatar
Brad King committed
448 449 450 451 452 453 454 455 456 457 458
        )
      list(APPEND files "${file}${stamp}")
    endif()
  endforeach()

  # Custom target to drive all update commands.
  add_custom_target(${target} ALL DEPENDS ${files})
endfunction()

function(ExternalData_expand_arguments target outArgsVar)
  # Replace DATA{} references with real arguments.
459
  set(data_regex "DATA{([^;{}\r\n]*)}")
Brad King's avatar
Brad King committed
460 461
  set(other_regex "([^D]|D[^A]|DA[^T]|DAT[^A]|DATA[^{])+|.")
  set(outArgs "")
462 463
  # This list expansion un-escapes semicolons in list element values so we
  # must re-escape them below anywhere a new list expansion will occur.
Brad King's avatar
Brad King committed
464 465
  foreach(arg IN LISTS ARGN)
    if("x${arg}" MATCHES "${data_regex}")
466 467
      # Re-escape in-value semicolons before expansion in foreach below.
      string(REPLACE ";" "\\;" tmp "${arg}")
Brad King's avatar
Brad King committed
468
      # Split argument into DATA{}-pieces and other pieces.
469
      string(REGEX MATCHALL "${data_regex}|${other_regex}" pieces "${tmp}")
Brad King's avatar
Brad King committed
470 471 472 473 474
      # Compose output argument with DATA{}-pieces replaced.
      set(outArg "")
      foreach(piece IN LISTS pieces)
        if("x${piece}" MATCHES "^x${data_regex}$")
          # Replace this DATA{}-piece with a file path.
475
          _ExternalData_arg("${target}" "${piece}" "${CMAKE_MATCH_1}" file)
Brad King's avatar
Brad King committed
476 477 478 479 480 481 482 483
          set(outArg "${outArg}${file}")
        else()
          # No replacement needed for this piece.
          set(outArg "${outArg}${piece}")
        endif()
      endforeach()
    else()
      # No replacements needed in this argument.
484
      set(outArg "${arg}")
Brad King's avatar
Brad King committed
485
    endif()
486 487 488
    # Re-escape in-value semicolons in resulting list.
    string(REPLACE ";" "\\;" outArg "${outArg}")
    list(APPEND outArgs "${outArg}")
Brad King's avatar
Brad King committed
489 490 491 492 493 494 495
  endforeach()
  set("${outArgsVar}" "${outArgs}" PARENT_SCOPE)
endfunction()

#-----------------------------------------------------------------------------
# Private helper interface

496 497
set(_ExternalData_REGEX_ALGO "MD5|SHA1|SHA224|SHA256|SHA384|SHA512")
set(_ExternalData_REGEX_EXT "md5|sha1|sha224|sha256|sha384|sha512")
Brad King's avatar
Brad King committed
498 499 500 501
set(_ExternalData_SELF "${CMAKE_CURRENT_LIST_FILE}")
get_filename_component(_ExternalData_SELF_DIR "${_ExternalData_SELF}" PATH)

function(_ExternalData_compute_hash var_hash algo file)
502 503
  if("${algo}" MATCHES "^${_ExternalData_REGEX_ALGO}$")
    file("${algo}" "${file}" hash)
Brad King's avatar
Brad King committed
504 505 506 507 508 509 510
    set("${var_hash}" "${hash}" PARENT_SCOPE)
  else()
    message(FATAL_ERROR "Hash algorithm ${algo} unimplemented.")
  endif()
endfunction()

function(_ExternalData_random var)
511
  string(RANDOM LENGTH 6 random)
Brad King's avatar
Brad King committed
512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527
  set("${var}" "${random}" PARENT_SCOPE)
endfunction()

function(_ExternalData_exact_regex regex_var string)
  string(REGEX REPLACE "([][+.*()^])" "\\\\\\1" regex "${string}")
  set("${regex_var}" "${regex}" PARENT_SCOPE)
endfunction()

function(_ExternalData_atomic_write file content)
  _ExternalData_random(random)
  set(tmp "${file}.tmp${random}")
  file(WRITE "${tmp}" "${content}")
  file(RENAME "${tmp}" "${file}")
endfunction()

function(_ExternalData_link_content name var_ext)
528
  if("${ExternalData_LINK_CONTENT}" MATCHES "^(${_ExternalData_REGEX_ALGO})$")
Brad King's avatar
Brad King committed
529 530 531 532 533 534 535 536 537
    set(algo "${ExternalData_LINK_CONTENT}")
  else()
    message(FATAL_ERROR
      "Unknown hash algorithm specified by ExternalData_LINK_CONTENT:\n"
      "  ${ExternalData_LINK_CONTENT}")
  endif()
  _ExternalData_compute_hash(hash "${algo}" "${name}")
  get_filename_component(dir "${name}" PATH)
  set(staged "${dir}/.ExternalData_${algo}_${hash}")
538
  string(TOLOWER ".${algo}" ext)
Brad King's avatar
Brad King committed
539 540 541 542 543 544 545 546 547 548 549 550 551 552
  _ExternalData_atomic_write("${name}${ext}" "${hash}\n")
  file(RENAME "${name}" "${staged}")
  set("${var_ext}" "${ext}" PARENT_SCOPE)

  file(RELATIVE_PATH relname "${ExternalData_SOURCE_ROOT}" "${name}${ext}")
  message(STATUS "Linked ${relname} to ExternalData ${algo}/${hash}")
endfunction()

function(_ExternalData_arg target arg options var_file)
  # Separate data path from the options.
  string(REPLACE "," ";" options "${options}")
  list(GET options 0 data)
  list(REMOVE_AT options 0)

553 554 555 556 557
  # Interpret trailing slashes as directories.
  set(data_is_directory 0)
  if("x${data}" MATCHES "^x(.*)([/\\])$")
    set(data_is_directory 1)
    set(data "${CMAKE_MATCH_1}")
558 559
  endif()

Brad King's avatar
Brad King committed
560 561 562 563
  # Convert to full path.
  if(IS_ABSOLUTE "${data}")
    set(absdata "${data}")
  else()
564
    set(absdata "${CMAKE_CURRENT_SOURCE_DIR}/${data}")
Brad King's avatar
Brad King committed
565
  endif()
566
  get_filename_component(absdata "${absdata}" ABSOLUTE)
Brad King's avatar
Brad King committed
567 568 569 570 571 572 573 574 575 576 577 578 579

  # Convert to relative path under the source tree.
  if(NOT ExternalData_SOURCE_ROOT)
    set(ExternalData_SOURCE_ROOT "${CMAKE_SOURCE_DIR}")
  endif()
  set(top_src "${ExternalData_SOURCE_ROOT}")
  file(RELATIVE_PATH reldata "${top_src}" "${absdata}")
  if(IS_ABSOLUTE "${reldata}" OR "${reldata}" MATCHES "^\\.\\./")
    message(FATAL_ERROR "Data file referenced by argument\n"
      "  ${arg}\n"
      "does not lie under the top-level source directory\n"
      "  ${top_src}\n")
  endif()
580 581 582 583 584 585 586
  if(data_is_directory AND NOT IS_DIRECTORY "${top_src}/${reldata}")
    message(FATAL_ERROR "Data directory referenced by argument\n"
      "  ${arg}\n"
      "corresponds to source tree path\n"
      "  ${reldata}\n"
      "that does not exist as a directory!")
  endif()
Brad King's avatar
Brad King committed
587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602
  if(NOT ExternalData_BINARY_ROOT)
    set(ExternalData_BINARY_ROOT "${CMAKE_BINARY_DIR}")
  endif()
  set(top_bin "${ExternalData_BINARY_ROOT}")

  # Handle in-source builds gracefully.
  if("${top_src}" STREQUAL "${top_bin}")
    if(ExternalData_LINK_CONTENT)
      message(WARNING "ExternalData_LINK_CONTENT cannot be used in-source")
      set(ExternalData_LINK_CONTENT 0)
    endif()
    set(top_same 1)
  endif()

  set(external "") # Entries external to the source tree.
  set(internal "") # Entries internal to the source tree.
603
  set(have_original ${data_is_directory})
604
  set(have_original_as_dir 0)
Brad King's avatar
Brad King committed
605 606

  # Process options.
607
  set(series_option "")
608
  set(recurse_option "")
Brad King's avatar
Brad King committed
609 610 611
  set(associated_files "")
  set(associated_regex "")
  foreach(opt ${options})
612 613 614
    # Regular expression to match associated files.
    if("x${opt}" MATCHES "^xREGEX:([^:/]+)$")
      list(APPEND associated_regex "${CMAKE_MATCH_1}")
615
    elseif(opt STREQUAL ":")
616 617
      # Activate series matching.
      set(series_option "${opt}")
618 619 620
    elseif(opt STREQUAL "RECURSE:")
      # Activate recursive matching in directories.
      set(recurse_option "${opt}")
Brad King's avatar
Brad King committed
621 622 623 624 625 626 627 628 629
    elseif("x${opt}" MATCHES "^[^][:/*?]+$")
      # Specific associated file.
      list(APPEND associated_files "${opt}")
    else()
      message(FATAL_ERROR "Unknown option \"${opt}\" in argument\n"
        "  ${arg}\n")
    endif()
  endforeach()

630
  if(series_option)
631 632 633
    if(data_is_directory)
      message(FATAL_ERROR "Series option \"${series_option}\" not allowed with directories.")
    endif()
634 635 636
    if(associated_files OR associated_regex)
      message(FATAL_ERROR "Series option \"${series_option}\" not allowed with associated files.")
    endif()
637 638 639
    if(recurse_option)
      message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.")
    endif()
640
    # Load a whole file series.
Brad King's avatar
Brad King committed
641
    _ExternalData_arg_series()
642 643 644 645 646 647 648 649 650
  elseif(data_is_directory)
    if(associated_files OR associated_regex)
      # Load listed/matching associated files in the directory.
      _ExternalData_arg_associated()
    else()
      message(FATAL_ERROR "Data directory referenced by argument\n"
        "  ${arg}\n"
        "must list associated files.")
    endif()
651
  else()
652 653 654
    if(recurse_option)
      message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.")
    endif()
655 656 657 658 659 660
    # Load the named data file.
    _ExternalData_arg_single()
    if(associated_files OR associated_regex)
      # Load listed/matching associated files.
      _ExternalData_arg_associated()
    endif()
Brad King's avatar
Brad King committed
661 662 663
  endif()

  if(NOT have_original)
664 665 666 667 668 669 670 671
    if(have_original_as_dir)
      set(msg_kind FATAL_ERROR)
      set(msg "that is directory instead of a file!")
    else()
      set(msg_kind AUTHOR_WARNING)
      set(msg "that does not exist as a file (with or without an extension)!")
    endif()
    message(${msg_kind} "Data file referenced by argument\n"
Brad King's avatar
Brad King committed
672 673 674
      "  ${arg}\n"
      "corresponds to source tree path\n"
      "  ${reldata}\n"
675
      "${msg}")
Brad King's avatar
Brad King committed
676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692
  endif()

  if(external)
    # Make the series available in the build tree.
    set_property(GLOBAL APPEND PROPERTY
      _ExternalData_${target}_FETCH "${external}")
    set_property(GLOBAL APPEND PROPERTY
      _ExternalData_${target}_LOCAL "${internal}")
    set("${var_file}" "${top_bin}/${reldata}" PARENT_SCOPE)
  else()
    # The whole series is in the source tree.
    set("${var_file}" "${top_src}/${reldata}" PARENT_SCOPE)
  endif()
endfunction()

macro(_ExternalData_arg_associated)
  # Associated files lie in the same directory.
693 694 695 696 697
  if(data_is_directory)
    set(reldir "${reldata}")
  else()
    get_filename_component(reldir "${reldata}" PATH)
  endif()
Brad King's avatar
Brad King committed
698 699 700 701
  if(reldir)
    set(reldir "${reldir}/")
  endif()
  _ExternalData_exact_regex(reldir_regex "${reldir}")
702 703 704 705 706 707
  if(recurse_option)
    set(glob GLOB_RECURSE)
    set(reldir_regex "${reldir_regex}(.+/)?")
  else()
    set(glob GLOB)
  endif()
Brad King's avatar
Brad King committed
708 709 710 711

  # Find files named explicitly.
  foreach(file ${associated_files})
    _ExternalData_exact_regex(file_regex "${file}")
712
    _ExternalData_arg_find_files(${glob} "${reldir}${file}"
713
      "${reldir_regex}${file_regex}")
Brad King's avatar
Brad King committed
714 715 716 717 718 719 720 721 722
  endforeach()

  # Find files matching the given regular expressions.
  set(all "")
  set(sep "")
  foreach(regex ${associated_regex})
    set(all "${all}${sep}${reldir_regex}${regex}")
    set(sep "|")
  endforeach()
723
  _ExternalData_arg_find_files(${glob} "${reldir}" "${all}")
Brad King's avatar
Brad King committed
724 725 726 727 728
endmacro()

macro(_ExternalData_arg_single)
  # Match only the named data by itself.
  _ExternalData_exact_regex(data_regex "${reldata}")
729
  _ExternalData_arg_find_files(GLOB "${reldata}" "${data_regex}")
Brad King's avatar
Brad King committed
730 731 732 733
endmacro()

macro(_ExternalData_arg_series)
  # Configure series parsing and matching.
734 735 736
  set(series_parse_prefix "")
  set(series_parse_number "\\1")
  set(series_parse_suffix "\\2")
Brad King's avatar
Brad King committed
737
  if(ExternalData_SERIES_PARSE)
738 739 740 741 742 743 744
    if(ExternalData_SERIES_PARSE_NUMBER AND ExternalData_SERIES_PARSE_SUFFIX)
      if(ExternalData_SERIES_PARSE_PREFIX)
        set(series_parse_prefix "\\${ExternalData_SERIES_PARSE_PREFIX}")
      endif()
      set(series_parse_number "\\${ExternalData_SERIES_PARSE_NUMBER}")
      set(series_parse_suffix "\\${ExternalData_SERIES_PARSE_SUFFIX}")
    elseif(NOT "x${ExternalData_SERIES_PARSE}" MATCHES "^x\\([^()]*\\)\\([^()]*\\)\\$$")
Brad King's avatar
Brad King committed
745 746 747 748
      message(FATAL_ERROR
        "ExternalData_SERIES_PARSE is set to\n"
        "  ${ExternalData_SERIES_PARSE}\n"
        "which is not of the form\n"
749 750 751 752 753 754
        "  (<number>)(<suffix>)$\n"
        "Fix the regular expression or set variables\n"
        "  ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any\n"
        "  ExternalData_SERIES_PARSE_NUMBER = <number> regex group number\n"
        "  ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number\n"
        )
Brad King's avatar
Brad King committed
755 756 757
    endif()
    set(series_parse "${ExternalData_SERIES_PARSE}")
  else()
758
    set(series_parse "([0-9]*)(\\.[^./]*)$")
Brad King's avatar
Brad King committed
759 760 761 762
  endif()
  if(ExternalData_SERIES_MATCH)
    set(series_match "${ExternalData_SERIES_MATCH}")
  else()
763
    set(series_match "[_.-]?[0-9]*")
Brad King's avatar
Brad King committed
764 765 766
  endif()

  # Parse the base, number, and extension components of the series.
767
  string(REGEX REPLACE "${series_parse}" "${series_parse_prefix};${series_parse_number};${series_parse_suffix}" tuple "${reldata}")
Brad King's avatar
Brad King committed
768 769 770 771 772 773 774 775 776 777 778 779 780
  list(LENGTH tuple len)
  if(NOT "${len}" EQUAL 3)
    message(FATAL_ERROR "Data file referenced by argument\n"
      "  ${arg}\n"
      "corresponds to path\n"
      "  ${reldata}\n"
      "that does not match regular expression\n"
      "  ${series_parse}")
  endif()
  list(GET tuple 0 relbase)
  list(GET tuple 2 ext)

  # Glob files that might match the series.
781
  # Then match base, number, and extension.
Brad King's avatar
Brad King committed
782 783
  _ExternalData_exact_regex(series_base "${relbase}")
  _ExternalData_exact_regex(series_ext "${ext}")
784
  _ExternalData_arg_find_files(GLOB "${relbase}*${ext}"
Brad King's avatar
Brad King committed
785 786 787
    "${series_base}${series_match}${series_ext}")
endmacro()

788 789 790 791 792
function(_ExternalData_arg_find_files glob pattern regex)
  cmake_policy(PUSH)
  cmake_policy(SET CMP0009 NEW)
  file(${glob} globbed RELATIVE "${top_src}" "${top_src}/${pattern}*")
  cmake_policy(POP)
Brad King's avatar
Brad King committed
793
  foreach(entry IN LISTS globbed)
794
    if("x${entry}" MATCHES "^x(.*)(\\.(${_ExternalData_REGEX_EXT}))$")
795 796 797 798 799 800
      set(relname "${CMAKE_MATCH_1}")
      set(alg "${CMAKE_MATCH_2}")
    else()
      set(relname "${entry}")
      set(alg "")
    endif()
801 802 803
    if("x${relname}" MATCHES "^x${regex}$" # matches
        AND NOT "x${relname}" MATCHES "(^x|/)\\.ExternalData_" # not staged obj
        )
804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821
      if(IS_DIRECTORY "${top_src}/${entry}")
        if("${relname}" STREQUAL "${reldata}")
          set(have_original_as_dir 1)
        endif()
      else()
        set(name "${top_src}/${relname}")
        set(file "${top_bin}/${relname}")
        if(alg)
          list(APPEND external "${file}|${name}|${alg}")
        elseif(ExternalData_LINK_CONTENT)
          _ExternalData_link_content("${name}" alg)
          list(APPEND external "${file}|${name}|${alg}")
        elseif(NOT top_same)
          list(APPEND internal "${file}|${name}")
        endif()
        if("${relname}" STREQUAL "${reldata}")
          set(have_original 1)
        endif()
Brad King's avatar
Brad King committed
822 823 824 825 826 827
      endif()
    endif()
  endforeach()
  set(external "${external}" PARENT_SCOPE)
  set(internal "${internal}" PARENT_SCOPE)
  set(have_original "${have_original}" PARENT_SCOPE)
828
  set(have_original_as_dir "${have_original_as_dir}" PARENT_SCOPE)
Brad King's avatar
Brad King committed
829 830 831 832 833 834 835 836 837 838 839 840
endfunction()

#-----------------------------------------------------------------------------
# Private script mode interface

if(CMAKE_GENERATOR OR NOT ExternalData_ACTION)
  return()
endif()

if(ExternalData_CONFIG)
  include(${ExternalData_CONFIG})
endif()
841 842 843
if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES)
  message(FATAL_ERROR
    "Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!")
Brad King's avatar
Brad King committed
844 845 846 847 848 849 850 851
endif()

function(_ExternalData_link_or_copy src dst)
  # Create a temporary file first.
  get_filename_component(dst_dir "${dst}" PATH)
  file(MAKE_DIRECTORY "${dst_dir}")
  _ExternalData_random(random)
  set(tmp "${dst}.tmp${random}")
852
  if(UNIX AND NOT ExternalData_NO_SYMLINKS)
Brad King's avatar
Brad King committed
853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881
    # Create a symbolic link.
    set(tgt "${src}")
    if(relative_top)
      # Use relative path if files are close enough.
      file(RELATIVE_PATH relsrc "${relative_top}" "${src}")
      file(RELATIVE_PATH relfile "${relative_top}" "${dst}")
      if(NOT IS_ABSOLUTE "${relsrc}" AND NOT "${relsrc}" MATCHES "^\\.\\./" AND
          NOT IS_ABSOLUTE "${reldst}" AND NOT "${reldst}" MATCHES "^\\.\\./")
        file(RELATIVE_PATH tgt "${dst_dir}" "${src}")
      endif()
    endif()
    execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink "${tgt}" "${tmp}" RESULT_VARIABLE result)
  else()
    # Create a copy.
    execute_process(COMMAND "${CMAKE_COMMAND}" -E copy "${src}" "${tmp}" RESULT_VARIABLE result)
  endif()
  if(result)
    file(REMOVE "${tmp}")
    message(FATAL_ERROR "Failed to create\n  ${tmp}\nfrom\n  ${obj}")
  endif()

  # Atomically create/replace the real destination.
  file(RENAME "${tmp}" "${dst}")
endfunction()

function(_ExternalData_download_file url file err_var msg_var)
  set(retry 3)
  while(retry)
    math(EXPR retry "${retry} - 1")
882 883 884 885
    if(ExternalData_TIMEOUT_INACTIVITY)
      set(inactivity_timeout INACTIVITY_TIMEOUT ${ExternalData_TIMEOUT_INACTIVITY})
    elseif(NOT "${ExternalData_TIMEOUT_INACTIVITY}" EQUAL 0)
      set(inactivity_timeout INACTIVITY_TIMEOUT 60)
Brad King's avatar
Brad King committed
886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917
    else()
      set(inactivity_timeout "")
    endif()
    if(ExternalData_TIMEOUT_ABSOLUTE)
      set(absolute_timeout TIMEOUT ${ExternalData_TIMEOUT_ABSOLUTE})
    elseif(NOT "${ExternalData_TIMEOUT_ABSOLUTE}" EQUAL 0)
      set(absolute_timeout TIMEOUT 300)
    else()
      set(absolute_timeout "")
    endif()
    file(DOWNLOAD "${url}" "${file}" STATUS status LOG log ${inactivity_timeout} ${absolute_timeout} SHOW_PROGRESS)
    list(GET status 0 err)
    list(GET status 1 msg)
    if(err)
      if("${msg}" MATCHES "HTTP response code said error" AND
          "${log}" MATCHES "error: 503")
        set(msg "temporarily unavailable")
      endif()
    elseif("${log}" MATCHES "\nHTTP[^\n]* 503")
      set(err TRUE)
      set(msg "temporarily unavailable")
    endif()
    if(NOT err OR NOT "${msg}" MATCHES "partial|timeout|temporarily")
      break()
    elseif(retry)
      message(STATUS "[download terminated: ${msg}, retries left: ${retry}]")
    endif()
  endwhile()
  set("${err_var}" "${err}" PARENT_SCOPE)
  set("${msg_var}" "${msg}" PARENT_SCOPE)
endfunction()

918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941
function(_ExternalData_custom_fetch key loc file err_var msg_var)
  if(NOT ExternalData_CUSTOM_SCRIPT_${key})
    set(err 1)
    set(msg "No ExternalData_CUSTOM_SCRIPT_${key} set!")
  elseif(NOT EXISTS "${ExternalData_CUSTOM_SCRIPT_${key}}")
    set(err 1)
    set(msg "No '${ExternalData_CUSTOM_SCRIPT_${key}}' exists!")
  else()
    set(ExternalData_CUSTOM_LOCATION "${loc}")
    set(ExternalData_CUSTOM_FILE "${file}")
    unset(ExternalData_CUSTOM_ERROR)
    include("${ExternalData_CUSTOM_SCRIPT_${key}}")
    if(DEFINED ExternalData_CUSTOM_ERROR)
      set(err 1)
      set(msg "${ExternalData_CUSTOM_ERROR}")
    else()
      set(err 0)
      set(msg "no error")
    endif()
  endif()
  set("${err_var}" "${err}" PARENT_SCOPE)
  set("${msg_var}" "${msg}" PARENT_SCOPE)
endfunction()

Brad King's avatar
Brad King committed
942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963
function(_ExternalData_download_object name hash algo var_obj)
  # Search all object stores for an existing object.
  foreach(dir ${ExternalData_OBJECT_STORES})
    set(obj "${dir}/${algo}/${hash}")
    if(EXISTS "${obj}")
      message(STATUS "Found object: \"${obj}\"")
      set("${var_obj}" "${obj}" PARENT_SCOPE)
      return()
    endif()
  endforeach()

  # Download object to the first store.
  list(GET ExternalData_OBJECT_STORES 0 store)
  set(obj "${store}/${algo}/${hash}")

  _ExternalData_random(random)
  set(tmp "${obj}.tmp${random}")
  set(found 0)
  set(tried "")
  foreach(url_template IN LISTS ExternalData_URL_TEMPLATES)
    string(REPLACE "%(hash)" "${hash}" url_tmp "${url_template}")
    string(REPLACE "%(algo)" "${algo}" url "${url_tmp}")
964 965 966 967 968 969 970 971 972 973
    if(url MATCHES "^(.*)%\\(algo:([A-Za-z_][A-Za-z0-9_]*)\\)(.*)$")
      set(lhs "${CMAKE_MATCH_1}")
      set(key "${CMAKE_MATCH_2}")
      set(rhs "${CMAKE_MATCH_3}")
      if(DEFINED ExternalData_URL_ALGO_${algo}_${key})
        set(url "${lhs}${ExternalData_URL_ALGO_${algo}_${key}}${rhs}")
      else()
        set(url "${lhs}${algo}${rhs}")
      endif()
    endif()
Brad King's avatar
Brad King committed
974
    message(STATUS "Fetching \"${url}\"")
975 976 977 978 979
    if(url MATCHES "^ExternalDataCustomScript://([A-Za-z_][A-Za-z0-9_]*)/(.*)$")
      _ExternalData_custom_fetch("${CMAKE_MATCH_1}" "${CMAKE_MATCH_2}" "${tmp}" err errMsg)
    else()
      _ExternalData_download_file("${url}" "${tmp}" err errMsg)
    endif()
Brad King's avatar
Brad King committed
980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008
    set(tried "${tried}\n  ${url}")
    if(err)
      set(tried "${tried} (${errMsg})")
    else()
      # Verify downloaded object.
      _ExternalData_compute_hash(dl_hash "${algo}" "${tmp}")
      if("${dl_hash}" STREQUAL "${hash}")
        set(found 1)
        break()
      else()
        set(tried "${tried} (wrong hash ${algo}=${dl_hash})")
        if("$ENV{ExternalData_DEBUG_DOWNLOAD}" MATCHES ".")
          file(RENAME "${tmp}" "${store}/${algo}/${dl_hash}")
        endif()
      endif()
    endif()
    file(REMOVE "${tmp}")
  endforeach()

  get_filename_component(dir "${name}" PATH)
  set(staged "${dir}/.ExternalData_${algo}_${hash}")

  if(found)
    file(RENAME "${tmp}" "${obj}")
    message(STATUS "Downloaded object: \"${obj}\"")
  elseif(EXISTS "${staged}")
    set(obj "${staged}")
    message(STATUS "Staged object: \"${obj}\"")
  else()
1009 1010 1011
    if(NOT tried)
      set(tried "\n  (No ExternalData_URL_TEMPLATES given)")
    endif()
Brad King's avatar
Brad King committed
1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027
    message(FATAL_ERROR "Object ${algo}=${hash} not found at:${tried}")
  endif()

  set("${var_obj}" "${obj}" PARENT_SCOPE)
endfunction()

if("${ExternalData_ACTION}" STREQUAL "fetch")
  foreach(v ExternalData_OBJECT_STORES file name ext)
    if(NOT DEFINED "${v}")
      message(FATAL_ERROR "No \"-D${v}=\" value provided!")
    endif()
  endforeach()

  file(READ "${name}${ext}" hash)
  string(STRIP "${hash}" hash)

1028 1029
  if("${ext}" MATCHES "^\\.(${_ExternalData_REGEX_EXT})$")
    string(TOUPPER "${CMAKE_MATCH_1}" algo)
Brad King's avatar
Brad King committed
1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066
  else()
    message(FATAL_ERROR "Unknown hash algorithm extension \"${ext}\"")
  endif()

  _ExternalData_download_object("${name}" "${hash}" "${algo}" obj)

  # Check if file already corresponds to the object.
  set(stamp "${ext}-stamp")
  set(file_up_to_date 0)
  if(EXISTS "${file}" AND EXISTS "${file}${stamp}")
    file(READ "${file}${stamp}" f_hash)
    string(STRIP "${f_hash}" f_hash)
    if("${f_hash}" STREQUAL "${hash}")
      #message(STATUS "File already corresponds to object")
      set(file_up_to_date 1)
    endif()
  endif()

  if(file_up_to_date)
    # Touch the file to convince the build system it is up to date.
    execute_process(COMMAND "${CMAKE_COMMAND}" -E touch "${file}")
  else()
    _ExternalData_link_or_copy("${obj}" "${file}")
  endif()

  # Atomically update the hash/timestamp file to record the object referenced.
  _ExternalData_atomic_write("${file}${stamp}" "${hash}\n")
elseif("${ExternalData_ACTION}" STREQUAL "local")
  foreach(v file name)
    if(NOT DEFINED "${v}")
      message(FATAL_ERROR "No \"-D${v}=\" value provided!")
    endif()
  endforeach()
  _ExternalData_link_or_copy("${name}" "${file}")
else()
  message(FATAL_ERROR "Unknown ExternalData_ACTION=[${ExternalData_ACTION}]")
endif()