GitLab

Report the underlying system error.

This avoids spawning a new CMake command which isn't necessary.

If more than one content link references the same object, the build
system may launch multiple download processes for the same object
concurrently.  Use whichever one finishes first, and discard the others.

Without this, we replace the objects and use the last finisher instead
of the first.  This is okay on non-Windows platforms where `rename(2)`
gives reliable atomic replacement.  However, on Windows platforms and
NTFS this is less reliable.  I've observed `MoveFileEx` somehow cause
another process to get `ERROR_SHARING_VIOLATION` when attempting to read
the destination file.  We may be able to improve the `file(RENAME)`
implementation on modern Windows 10 versions, but for ExternalData's use
case it is simpler to just not replace existing objects.

Issue: #19715

During CI builds (at least), download progress is just noise. Allow it
to be suppressed.

Default to `OFF` for Ninja due to the tool's behavior of not showing
output until a command is complete (which makes any progress reporting
of little use) and `ON` otherwise.

Automate with:

find Modules -type f -print0 | xargs -0 perl -i -0pe \
  's/set\(([a-zA-Z0-9_\$\{\}]+)(\s+)"\$\{\1\}([^"])/string(APPEND \1\2"\3/g'

Matt McCormick authored Oct 12, 2016 and Brad King committed Oct 18, 2016

Add support for projects to have `Data.txt.md5` *and* `Data.txt.sha512`
where the content links hold hashes for the same file.  Check all
`ExternalData_URL_TEMPLATES` entries in order for all available hashes.
The data acquisition is considered a failure if none of the available
URL resources has any of the given hashes.  This makes it possible to
have multiple data server resources where all servers do not support all
hashing algorithms.

Per-source copyright/license notice headers that spell out copyright holder
names and years are hard to maintain and often out-of-date or plain wrong.
Precise contributor information is already maintained automatically by the
version control tool.  Ultimately it is the receiver of a file who is
responsible for determining its licensing status, and per-source notices are
merely a convenience.  Therefore it is simpler and more accurate for
each source to have a generic notice of the license name and references to
more detailed information on copyright holders and full license terms.

Our `Copyright.txt` file now contains a list of Contributors whose names
appeared source-level copyright notices.  It also references version control
history for more precise information.  Therefore we no longer need to spell
out the list of Contributors in each source file notice.

Replace CMake per-source copyright/license notice headers with a short
description of the license and lin...

If multiple ExternalData_Target_Add calls generate the same output file
then we need to avoid calling add_custom_command multiple times with
that output.  This was already done within a single target by setting a
variable in the local function scope.  This will not be visible in other
calls though so we need to use a directory property instead to prevent
adding a custom command multiple times for one output in a directory.

Normally it is not safe to have multiple custom commands that produce
the same output file across multiple independent targets, but since we
use atomic replacement of outputs the resulting races should not be a
problem.  For the convenience of projects, tolerate this instead of
diagnosing it.  In particular, we previously allowed up to two copies
of the custom command in one directory because CMake has a fallback
from MAIN_DEPENDENCY to an `<output>.rule` file.

While at it, add a note to the documentation that typically only one
external data target should be needed for a project.
Reported-by: David Manthey <david.manthey@kitware.com>

Automate with:

find Modules -type f -print0 | xargs -0 perl -i -0pe \
's/set\(([a-zA-Z0-9_]+)(\s+)"\$\{\1\}([^"])/string(APPEND \1\2"\3/g'

Do not disclose credentials when printing URIs used to fetch external resources.

Add an ExternalData_NO_SYMLINKS to enable use of copies instead of
symlinks to populate the real data files behind a DATA{} reference.
This will be useful on UNIX-like systems when the underlying filesystem
does not actually support symbolic links.
Suggested-by: Matt McCormick <matt.mccormick@kitware.com>

Extend the ``DATA{Dir/,...}`` syntax with a new ``RECURSE:`` option
to enable recursive matching of associated files.  This will allow
an entire directory tree of data to be referenced at once.

Extend the _ExternalData_arg_find_files signature with an option to
specify the kind of file(GLOB) operation to be performed.  Set
CMP0009 to NEW so that GLOB_RECURSE does not follow symlinks.

Allow URL templates to contain a %(algo:<key>) placeholder that is
replaced by mapping the canonical hash algorithm name through a map
defined by the <key>.

Extend the Module.ExternalData test to cover the behavior.
Extend the RunCMake.ExternalData test to cover error cases.

Add support for a special URL template to map the fetch operation
to a project-specified .cmake script insead of using file(DOWNLOAD).

Extend the Module.ExternalData test to cover the behavior.
Extend the RunCMake.ExternalData test to cover error cases.

Move the basic DATA{} description to a section just before the
file series description.  Move all sections on referencing files
into subsections of a common "Referencing Files" section.

Subsume example usage into the introduction since it gives a
high-level starting point to understand the rest of the docs.

Put example usage just after the intro.  Put hash algos last.
Put variables right after functions since they both cover the
module API.

Extend the "Module Variables" section to list variables
previously mentioned only in text.

Also explicitly mark functions and variables.

Use a bracket comment to hold the documentation instead of a block of
line comments.  This will make further updates easier.

Let clients do their own research on the current strength of each
hash algorithm to choose what is best for their needs.
Suggested-by: Nico Schlömer <nico.schloemer@gmail.com>

When the primary source tree path named by a DATA{} reference does not
exist, produce an AUTHOR_WARNING instead of a FATAL_ERROR.  This is
useful when writing a new DATA{} reference to a test reference output
that has not been created yet.  This way the developer can run the test,
manually verify the output, and then copy it into place to provide the
reference and eliminate the warning.

If the named source tree path is expected to be a file but exists as a
directory, we still need to produce a FATAL_ERROR.

The matches have already been calculated and can simply be taken from
CMAKE_MATCH_n variables. This avoids multiple compilations of the same or very
similar regular expressions.

Allow ExternalData_URL_TEMPLATES to be empty if a value for
ExternalData_OBJECT_STORES is provided.  Assume in this use case that
the object stores will already contain all needed objects.  Extend the
Module.ExternalData test to cover this case (all objects in stores).
Extend the RunCMake.ExternalData test to cover the non-failure message
case when stores are provided without URL templates.

In the ExternalData_URL_TEMPLATES example, add an entry showing how to
use the "file://" protocol with a network filesystem path.

Manually revise the .rst format of the documentation.  Use inline
literal quotes appropriately in paragraph text.  Move the :: literal
block openers to the end of the preceding paragraphs.  Convert the
command signature documentation and examples to cmake code-block
directives.

Kitware Robot authored Oct 15, 2013 and Brad King committed Oct 15, 2013

Run the convert-help.bash script to convert documentation:

 ./convert-help.bash "/path/to/CMake-build/bin"

Then remove it.

The ExternalData_LINK_CONTENT option tells ExternalData to convert real
data files it finds into content links and to "stage" the original
content in a ".ExternalData_<algo>_<hash>" file.  However, after a data
object has been staged it is possible that a user-provided pattern in
the "REGEX:" option will later match the staged object file.  We must
not process staged object files even when a user pattern matches them.

Fix the implementation to not match a staged object file as a normal
data file for conversion.  Extend the RunCMake.ExternalData test to
cover this case.

The CMake language implicitly flattens lists so a ";" in a list element
must be escaped with a backslash.  List expansion removes backslashes
escaping semicolons to leave raw semicolons in the values.  Teach
ExternalData_Add_Test and ExternalData_Expand_Arguments to re-escape
semicolons found in list elements so the resulting argument lists work
as if constructed directly by the set() command.

For example:

  ExternalData_Add_Test(Data NAME test1 COMMAND ... "a\\;b")
  ExternalData_Expand_Arguments(Data args2 "c\\;d")
  add_test(NAME test2 COMMAND ... ${args2})

should be equivalent to

  set(args1 "a\\;b")
  add_test(NAME test1 COMMAND ... ${args1})
  set(args2 "c\\;d")
  add_test(NAME test2 COMMAND ... ${args2})

which is equivalent to

  add_test(NAME test1 COMMAND ... "a;b")
  add_test(NAME test2 COMMAND ... "c;d")

Note that it is not possible to make ExternalData_Add_Test act exactly
like add_test when quoted arguments contain semicolons because the CMake
language flattens lists when constructing function ARGN values.  This
re-escape approach at least allows test arguments to have semicolons.

While at it, teach ExternalData APIs to not transform "DATA{...;...}"
arguments because the contained semicolons are non-sensical.
Suggested-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>

Each data file to be created in the build tree corresponds one-to-one with
a raw file or content link in the source tree.  Use the MAIN_DEPENDENCY of
add_custom_command to attach the build rule to the source tree file.  This
looks much nicer in the IDE project file browser and avoids ".rule" files.

Relative path components need to be normalized out even if they appear
in the middle of a caller-supplied string.

Update the Module.ExternalData and RunCMake.ExternalData tests to cover
some of them.

Use private global variables _ExternalData_REGEX_(ALGO|EXT) to match the
possible hash algorithm names and extensions in regular expressions.
Use "file(<algo>)" instead of "cmake -E md5sum" to compute hashes
without a child process and to support more hash algorithms.