data.md 11.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281
VTK Data
========

This page documents how to add test data while developing VTK with [Git][].
See the [README](README.md) for more information.

[Git]: http://git-scm.com

Setup
-----

The workflow below depends on local hooks to function properly.
Follow the main [developer setup instructions](develop.md#setup)
before proceeding.  In particular, run [SetupForDevelopment.sh][]:

    $ ./Utilities/SetupForDevelopment.sh

[SetupForDevelopment.sh]: /Utilities/SetupForDevelopment.sh

Workflow
--------

Our workflow for adding data integrates with our standard Git
[development process](develop.md).  Start by
[creating a topic](develop.md#create-a-topic).
Return here when you reach the "edit files" step.

These instructions follow a typical use case of adding a new
test with a baseline image.

### Add Test ###

1.  Write a new test, e.g.

        $ edit Some/Module/Testing/Cxx/MyTest.cxx

2.  Edit the corresponding `CMakeLists.txt` file:

        $ edit Some/Module/Testing/Cxx/CMakeLists.txt

    and add the test in a `vtk_add_test_...` call (which references
    baselines automatically).

3.  For tests not using such a call, reference the data file in an
    `ExternalData_add_test` call.  Specify the file inside `DATA{...}`
    using a path relative to the test directory:

        $ edit Some/Module/Testing/Cxx/CMakeLists.txt
        ExternalData_add_test(VTKData
          NAME ${vtk-module}Cxx-MyTest
          COMMAND ${vtk-module}CxxTests MyTest
                  ... -V DATA{../Data/Baseline/MyTest.png,:} ...
          )

Notes:

* If the data file references other data files, e.g. `.mhd -> .raw`,
  read the [ExternalData][] module documentation on "associated" files.
* Multiple baseline images and other series are handled automatically
  when the reference ends in the `,:` option.  Read [ExternalData][]
  module documentation for details.

[ExternalData]: /CMake/ExternalData.cmake

### Build and Run the Test ###

If you already have a data file, skip to the [next step](#add-data) to add it.
Otherwise, use the following steps to produce a test baseline image file.
We assume a build tree has been previously generated by CMake.

1.  Switch to the build tree:

        $ cd ../VTK-build

2.  Run CMake:

        $ cmake .

    Since we have not yet created the baseline image data file, CMake
    will warn that it does not exist but proceed to generate the test
    anyway.

3.  Build

        $ make

4.  Run the test

        $ ctest -R MyTest

    It will fail but place the baseline image in `Testing/Temporary`.

5.  Switch back to the source tree:

        $ cd ../VTK

### Add Data ###

Copy the data file into your local source tree.

    $ mkdir -p Some/Module/Testing/Data/Baseline
    $ cp ../VTK-build/Testing/Temporary/MyTest.png Some/Module/Testing/Data/Baseline

### Run CMake ###

1.  Switch to the build tree:

        $ cd ../VTK-build

2.  Run CMake:

        $ cmake .

    CMake will [move the original file](#externaldata).  Keep your own
    copy if necessary.  See [below](#recover-data-file) to recover the
    original file.

    During configuration CMake will display a message such as:

        Linked Some/Module/Testing/Data/Baseline/MyTest.png.md5 to ExternalData MD5/...

    This means that CMake converted the file into a data object referenced
    by a "content link" named like the original file but with a `.md5`
    extension.  CMake also [renamed the original file](#externaldata).

3.  Build

        $ make

    During the build, the [ExternalData][] module will make the data
    file available where the test expects to find it.

4.  Run the test

        $ ctest -R MyTest

    It should pass using the new data file.

5.  Switch back to the source tree:

        $ cd ../VTK

### Commit ###

Continue to [create the topic](develop.md#create-a-topic) and edit other
files as necessary.  Add the content link and commit it along with the
other changes:

    $ git add Some/Module/Testing/Data/Baseline/MyTest.png.md5
    $ git add Some/Module/Testing/Data/CMakeLists.txt
    $ git commit

The local `pre-commit` hook will display a message such as:

    Some/Module/Testing/Data/Baseline/MyTest.png.md5: Added content to Git at refs/data/MD5/...
    Some/Module/Testing/Data/Baseline/MyTest.png.md5: Added content to local store at .ExternalData/MD5/...
    Content link Some/Module/Testing/Data/Baseline/MyTest.png.md5 -> .ExternalData/MD5/...

This means that the pre-commit hook recognized that the content link
references a new data object and [prepared it for upload](#pre-commit).

### Push ###

Follow the instructions to [share the topic](develop.md#share-a-topic).
When you push it to GitLab for review using

    $ git gitlab-push

part of the output will be of the form:

    *       ...:refs/data/...      [new branch]
    *       HEAD:refs/heads/my-topic  [new branch]
    Pushed refs/data/... and removed local ref.

This means that the `git-gitlab-push` script pushed the topic
and [uploaded the data](#git-gitlab-push) it references.

Options for `gitlab-push` include:
* `--dry-run`: Report push that would occur without actually doing it
* `--no-topic`: Push the data referenced by the topic but not the topic itself

Note: One must `git gitlab-push` from the same work tree as was used
to create the commit.  Do not `git push` to another computer first and
try to push to GitLab from there because the data will not follow.

Building
--------

### Download ###

For the test data to be downloaded and made available to the tests in
your build tree the `VTKData` target must be built.  One may build the
target directly, e.g. `make VTKData`, to obtain the data without a
complete build.  The output will be something like

    -- Fetching ".../ExternalData/MD5/..."
    -- [download 100% complete]
    -- Downloaded object: "VTK-build/ExternalData/Objects/MD5/..."

The downloaded files appear in `VTK-build/ExternalData` by default.

### Local Store ###

It is possible to configure one or more local ExternalData object
stores shared among multiple builds.  Configure for each build the
advanced cache entry `ExternalData_OBJECT_STORES` to a directory on
your local disk outside all build trees, e.g. `/home/user/.ExternalData`:

    $ cmake -DExternalData_OBJECT_STORES=/home/user/.ExternalData ../VTK

The [ExternalData][] module will store downloaded objects in the local
store instead of the build tree.  Once an object has been downloaded
by one build it will persist in the local store for re-use by other
builds without downloading again.

Discussion
----------

A VTK test data file is not stored in the main source tree under version
control.  Instead the source tree contains a "content link" that refers
to a data object by a hash of its content.  At build time the
[ExternalData][] module fetches data needed by enabled tests.
This allows arbitrarily large data to be added and removed without
bloating the version control history.

The above [workflow](#workflow) allows developers to add a new data file
almost as if committing it to the source tree.  The following subsections
discuss details of the workflow implementation.

### ExternalData ###

While [CMake runs](#run-cmake) the [ExternalData][] module evaluates
[DATA{} references](#add-test).  VTK [sets](/CMake/vtkExternalData.cmake)
the `ExternalData_LINK_CONTENT` option to `MD5` to enable automatic
conversion of raw data files into content links.  When the module detects
a real data file in the source tree it performs the following
transformation as specified in the module documentation:

* Compute the MD5 hash of the file
* Store the `${hash}` in a file with the original name plus `.md5`
* Rename the original file to `.ExternalData_MD5_${hash}`

The real data now sit in a file that we [tell Git to ignore](/.gitignore).
For example:

    $ cat Some/Module/Testing/Data/Baseline/.ExternalData_MD5_477e602800c18624d9bc7a32fa706b97 |md5sum
    477e602800c18624d9bc7a32fa706b97  -
    $ cat Some/Module/Testing/Data/Baseline/MyTest.png.md5
    477e602800c18624d9bc7a32fa706b97

#### Recover Data File ####

To recover the original file after running CMake but before committing,
undo the operation:

    $ cd Some/Module/Testing/Data/Baseline
    $ mv .ExternalData_MD5_$(cat MyTest.png.md5) MyTest.png

### pre-commit ###

While [committing](#commit) a new or modified content link the
[pre-commit](/Utilities/Scripts/pre-commit) hook moves the real data
object from the `.ExternalData_MD5_${hash}` file left by the
[ExternalData][] module to a local object repository stored in a
`.ExternalData` directory at the top of the source tree.

The hook also uses Git plumbing commands to store the data object
as a blob in the local Git repository.  The blob is not referenced
by the new commit but instead by `refs/data/MD5/${hash}`.
This keeps the blob alive in the local repository but does not add
it to the project history.  For example:

    $ git for-each-ref --format="%(refname)" refs/data
    refs/data/MD5/477e602800c18624d9bc7a32fa706b97
    $ git cat-file blob refs/data/MD5/477e602800c18624d9bc7a32fa706b97 | md5sum
    477e602800c18624d9bc7a32fa706b97  -

### git gitlab-push ###

The `git gitlab-push` command is actually an alias for the
[git-gitlab-push](/Utilities/GitSetup/git-gitlab-push) script.
282
In addition to pushing the topic branch to GitLab the script also detects
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298
content links added or modified by the commits in the topic.
It reads the data object hashes from the content links and looks for
matching `refs/data/` entries in the local Git repository.

The script pushes the matching data objects to your VTK GitLab fork.
For example:

    $ git gitlab-push --dry-run --no-topic
    *       refs/data/MD5/477e602800c18624d9bc7a32fa706b97:refs/data/MD5/477e602800c18624d9bc7a32fa706b97   [new branch]
    Pushed refs/data/MD5/477e602800c18624d9bc7a32fa706b97 and removed local ref.

A GitLab webhook that triggers whenever a topic branch is pushed checks
for `refs/data/` in your VTK GitLab fork, fetches them, erases the refs
from your fork, and uploads them to a location that we
[tell ExternalData to search](/CMake/vtkExternalData.cmake) at build time.

299 300 301 302 303 304 305
To verify that the data has been uploaded as expected, you may direct
a web browser to the location where ExternalData has uploaded the files.
For VTK, that location is currently
`http://www.vtk.org/files/ExternalData/MD5/XXXX` where `XXXX` is the
complete MD5 hash stored in the content link file (e.g., the text in
`MyTest.png.md5`).

306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337
### Publishing Data for an External Branch ###

The above [workflow](#workflow) works well for developers working on a
single machine to contribute changes directly to upstream VTK.  When
working in an external branch of VTK, perhaps during a long-term topic
development effort, data objects need to be published separately.

The workflow for adding data to an external branch of VTK is the same
as the above through the [commit](#commit) step, but diverges at the
[push](#push) step because one will push to a separate repository.
Our ExternalData infrastructure intentionally hides the real data files
from Git so only the content links (`.md5` files) will be pushed.
The real data objects will still be left in the `.ExternalData/MD5`
directory at the top of the VTK source tree by the
[pre-commit](#pre-commit) hook.

The `.ExternalData` directory must be published somewhere visible to
other machines that want to use the data, such as on a web server.
Once that is done then other machines can be told where to look for
the data, e.g.

    cmake ../VTK "-DExternalData_URL_TEMPLATES=https://username.github.io/VTK/ExternalData/%(algo)/%(hash)

In this example we assume the files are published on a [Github Pages][]
`gh-pages` branch in `username`'s fork of VTK.

Within the `gh-pages` branch the files are placed at
`ExternalData/MD5/$md5sum` where `$md5sum` is the MD5 hash of the content
(these are the same names they have in the `.ExternalData` directory in
the original source tree).

[Github Pages]: https://help.github.com/articles/creating-project-pages-manually