URL substitution for file(DOWNLOAD) and ExternalProject download methods
Some CMake operations need to download, clone, checkout, etc. from a URL. Often, these are tied to specific well-known locations, but sometimes this can lead to problems when the resource being referred to needs to be taken from somewhere else instead. For example, ExternalProject_Add()
and FetchContent_Declare()
typically include some sort of URL for the source of an archive to download or repository to clone code from. The user might prefer to make use of a more local mirror, or they might have their own fork that they want to use instead. Similarly, some projects call file(DOWNLOAD)
directly to obtain certain artefacts, but again the user might want to redirect such calls to find the file from somewhere else (e.g. already stored locally so they can work offline).
It would be useful if the user could provide a file that contained a set of URL substitutions that these commands would automatically and transparently apply. This would allow them to use projects unmodified and instead contain their redirections locally in a file they maintain themselves. They might also share such a "redirections" file between projects.
A proposed solution to this problem is to support a variable called URL_REDIRECTIONS_FILE
(working name for discussion). The file(DOWNLOAD)
, ExternalProject_Add()
and FetchContent_Declare()
commands (and any others we can identify as relevant) would check whether the URL they've been asked to use matches any redirections in that file and silently use the redirected URL instead if there is one. We should also provide a command that allows projects to request the resolved URL for any input URL, which would allow them to also make use of this functionality for custom purposes. Projects themselves should never specify or provide this redirections file, it would be intended as a user feature. Such a file would have the following requirements:
- Easy for humans to understand and edit, preferably using an existing file format. In keeping with other features already supported by CMake, JSON would have the strongest precedent for use here and would keep open the ability to extend the file schema in the future if needed. Alternatively, a basic line-based file might also be suitable, but it lacks the flexibility of supporting extended use cases (e.g. we might want to make some substitutions only apply in certain situations, to be captured by some sort of condition).
- Match URLs by regular expression and allow the substitutions to refer to match objects (
\1
and so on). - Clearly defined semantics if there are multiple matches (e.g. first one wins, or last one wins).
- Clearly defined semantics around case sensitivity of matches. For simplicity, we probably want to just assume all regular expressions are case sensitive, but we should spell this out in any documentation (since parts of URLs are technically not case-sensitive).
Other potential features, recommend not including these in a first iteration but need to be considered for design:
- Support for file inclusion so users could compose redirections with multiple files.
- Support for file uploads as well, not just downloads.