Commit 86589946 authored by Brad King's avatar Brad King Committed by Utkarsh Ayachit
Browse files

Documentation: Add VTK Git instructions

Port instructions from the http://www.vtk.org/Wiki/VTK/Git wiki page to
Markdown documents in the source tree.  Update the instructions to work
for the GitLab environment.
parent 052ee157
VTK Git Usage
=============
VTK version tracking and development is hosted by [Git](http://git-scm.com).
Please select a task for further instructions:
Main Tasks
----------
* [Install Git](http://public.kitware.com/Wiki/Git/Download) -
Git 1.6.6 or greater is preferred (required for development)
* [Download VTK](download.md) - Users start here
* [Develop VTK](develop.md) - Contributors start here
Other Tasks
-----------
* [Review Changes](https://gitlab.kitware.com/vtk/vtk/merge_requests) -
VTK GitLab Merge Requests
* [Test VTK](dashboard.md) - CDash client setup
* [Learn Git](http://public.kitware.com/Wiki/Git/Resources) -
Third-party documentation
Branches
--------
The upstream VTK repository has the following branches:
* `master`: Development (default)
* `release`: Maintenance of latest release
* `nightly-master`: Follows `master`, updated at `01:00 UTC`
* `hooks`: Local commit hooks
([placed](http://public.kitware.com/Wiki/Git/Hooks#Local) in `.git/hooks`)
* `dashboard`: Dashboard script ([setup](dashboard.md) a CDash client)
VTK Dashboard Scripts
=====================
This page documents how to use the VTK `dashboard` branch in [Git][].
See the [README](README.md) for more information.
[Git]: http://git-scm.com
Using the Dashboard Scripts
---------------------------
The `dashboard` branch contains a dashboard client helper script.
Use these commands to track it:
$ mkdir -p ~/Dashboards/VTKScripts
$ cd ~/Dashboards/VTKScripts
$ git init
$ git remote add -t dashboard origin https://gitlab.kitware.com/vtk/vtk.git
$ git pull origin
The `vtk_common.cmake` script contains setup instructions in its
top comments.
Update the `dashboard` branch to get the latest version of this
script by simply running:
$ git pull
Here is a link to the script as it appears today: [vtk_common.cmake][].
[vtk_common.cmake]: https://gitlab.kitware.com/vtk/vtk/tree/dashboard/vtk_common.cmake
Changing the Dashboard Scripts
------------------------------
If you find bugs in the hooks themselves or would like to add new features,
the can be edited in the usual Git manner:
$ git checkout -b my-topic-branch
Make your edits, test it, and commit the result. Create a patch file with:
$ git format-patch origin/dashboard
And email the results to the developer's mailing list.
VTK Data
========
This page documents how to add test data while developing VTK with [Git][].
See the [README](README.md) for more information.
[Git]: http://git-scm.com
Setup
-----
The workflow below depends on local hooks to function properly.
Follow the main [developer setup instructions](develop.md#setup)
before proceeding. In particular, run [SetupForDevelopment.sh][]:
$ ./Utilities/SetupForDevelopment.sh
[SetupForDevelopment.sh]: /Utilities/SetupForDevelopment.sh
Workflow
--------
Our workflow for adding data integrates with our standard Git
[development process](develop.md). Start by
[creating a topic](develop.md#create-a-topic).
Return here when you reach the "edit files" step.
These instructions follow a typical use case of adding a new
test with a baseline image.
### Add Test ###
1. Write a new test, e.g.
$ edit Some/Module/Testing/Cxx/MyTest.cxx
2. Edit the corresponding `CMakeLists.txt` file:
$ edit Some/Module/Testing/Cxx/CMakeLists.txt
and add the test in a `vtk_add_test_...` call (which references
baselines automatically).
3. For tests not using such a call, reference the data file in an
`ExternalData_add_test` call. Specify the file inside `DATA{...}`
using a path relative to the test directory:
$ edit Some/Module/Testing/Cxx/CMakeLists.txt
ExternalData_add_test(VTKData
NAME ${vtk-module}Cxx-MyTest
COMMAND ${vtk-module}CxxTests MyTest
... -V DATA{../Data/Baseline/MyTest.png,:} ...
)
Notes:
* If the data file references other data files, e.g. `.mhd -> .raw`,
read the [ExternalData][] module documentation on "associated" files.
* Multiple baseline images and other series are handled automatically
when the reference ends in the `,:` option. Read [ExternalData][]
module documentation for details.
[ExternalData]: /CMake/ExternalData.cmake
### Build and Run the Test ###
If you already have a data file, skip to the [next step](#add-data) to add it.
Otherwise, use the following steps to produce a test baseline image file.
We assume a build tree has been previously generated by CMake.
1. Switch to the build tree:
$ cd ../VTK-build
2. Run CMake:
$ cmake .
Since we have not yet created the baseline image data file, CMake
will warn that it does not exist but proceed to generate the test
anyway.
3. Build
$ make
4. Run the test
$ ctest -R MyTest
It will fail but place the baseline image in `Testing/Temporary`.
5. Switch back to the source tree:
$ cd ../VTK
### Add Data ###
Copy the data file into your local source tree.
$ mkdir -p Some/Module/Testing/Data/Baseline
$ cp ../VTK-build/Testing/Temporary/MyTest.png Some/Module/Testing/Data/Baseline
### Run CMake ###
1. Switch to the build tree:
$ cd ../VTK-build
2. Run CMake:
$ cmake .
CMake will [move the original file](#externaldata). Keep your own
copy if necessary. See [below](#recover-data-file) to recover the
original file.
During configuration CMake will display a message such as:
Linked Some/Module/Testing/Data/Baseline/MyTest.png.md5 to ExternalData MD5/...
This means that CMake converted the file into a data object referenced
by a "content link" named like the original file but with a `.md5`
extension. CMake also [renamed the original file](#externaldata).
3. Build
$ make
During the build, the [ExternalData][] module will make the data
file available where the test expects to find it.
4. Run the test
$ ctest -R MyTest
It should pass using the new data file.
5. Switch back to the source tree:
$ cd ../VTK
### Commit ###
Continue to [create the topic](develop.md#create-a-topic) and edit other
files as necessary. Add the content link and commit it along with the
other changes:
$ git add Some/Module/Testing/Data/Baseline/MyTest.png.md5
$ git add Some/Module/Testing/Data/CMakeLists.txt
$ git commit
The local `pre-commit` hook will display a message such as:
Some/Module/Testing/Data/Baseline/MyTest.png.md5: Added content to Git at refs/data/MD5/...
Some/Module/Testing/Data/Baseline/MyTest.png.md5: Added content to local store at .ExternalData/MD5/...
Content link Some/Module/Testing/Data/Baseline/MyTest.png.md5 -> .ExternalData/MD5/...
This means that the pre-commit hook recognized that the content link
references a new data object and [prepared it for upload](#pre-commit).
### Push ###
Follow the instructions to [share the topic](develop.md#share-a-topic).
When you push it to GitLab for review using
$ git gitlab-push
part of the output will be of the form:
* ...:refs/data/... [new branch]
* HEAD:refs/heads/my-topic [new branch]
Pushed refs/data/... and removed local ref.
This means that the `git-gitlab-push` script pushed the topic
and [uploaded the data](#git-gitlab-push) it references.
Options for `gitlab-push` include:
* `--dry-run`: Report push that would occur without actually doing it
* `--no-topic`: Push the data referenced by the topic but not the topic itself
Note: One must `git gitlab-push` from the same work tree as was used
to create the commit. Do not `git push` to another computer first and
try to push to GitLab from there because the data will not follow.
Building
--------
### Download ###
For the test data to be downloaded and made available to the tests in
your build tree the `VTKData` target must be built. One may build the
target directly, e.g. `make VTKData`, to obtain the data without a
complete build. The output will be something like
-- Fetching ".../ExternalData/MD5/..."
-- [download 100% complete]
-- Downloaded object: "VTK-build/ExternalData/Objects/MD5/..."
The downloaded files appear in `VTK-build/ExternalData` by default.
### Local Store ###
It is possible to configure one or more local ExternalData object
stores shared among multiple builds. Configure for each build the
advanced cache entry `ExternalData_OBJECT_STORES` to a directory on
your local disk outside all build trees, e.g. `/home/user/.ExternalData`:
$ cmake -DExternalData_OBJECT_STORES=/home/user/.ExternalData ../VTK
The [ExternalData][] module will store downloaded objects in the local
store instead of the build tree. Once an object has been downloaded
by one build it will persist in the local store for re-use by other
builds without downloading again.
Discussion
----------
A VTK test data file is not stored in the main source tree under version
control. Instead the source tree contains a "content link" that refers
to a data object by a hash of its content. At build time the
[ExternalData][] module fetches data needed by enabled tests.
This allows arbitrarily large data to be added and removed without
bloating the version control history.
The above [workflow](#workflow) allows developers to add a new data file
almost as if committing it to the source tree. The following subsections
discuss details of the workflow implementation.
### ExternalData ###
While [CMake runs](#run-cmake) the [ExternalData][] module evaluates
[DATA{} references](#add-test). VTK [sets](/CMake/vtkExternalData.cmake)
the `ExternalData_LINK_CONTENT` option to `MD5` to enable automatic
conversion of raw data files into content links. When the module detects
a real data file in the source tree it performs the following
transformation as specified in the module documentation:
* Compute the MD5 hash of the file
* Store the `${hash}` in a file with the original name plus `.md5`
* Rename the original file to `.ExternalData_MD5_${hash}`
The real data now sit in a file that we [tell Git to ignore](/.gitignore).
For example:
$ cat Some/Module/Testing/Data/Baseline/.ExternalData_MD5_477e602800c18624d9bc7a32fa706b97 |md5sum
477e602800c18624d9bc7a32fa706b97 -
$ cat Some/Module/Testing/Data/Baseline/MyTest.png.md5
477e602800c18624d9bc7a32fa706b97
#### Recover Data File ####
To recover the original file after running CMake but before committing,
undo the operation:
$ cd Some/Module/Testing/Data/Baseline
$ mv .ExternalData_MD5_$(cat MyTest.png.md5) MyTest.png
### pre-commit ###
While [committing](#commit) a new or modified content link the
[pre-commit](/Utilities/Scripts/pre-commit) hook moves the real data
object from the `.ExternalData_MD5_${hash}` file left by the
[ExternalData][] module to a local object repository stored in a
`.ExternalData` directory at the top of the source tree.
The hook also uses Git plumbing commands to store the data object
as a blob in the local Git repository. The blob is not referenced
by the new commit but instead by `refs/data/MD5/${hash}`.
This keeps the blob alive in the local repository but does not add
it to the project history. For example:
$ git for-each-ref --format="%(refname)" refs/data
refs/data/MD5/477e602800c18624d9bc7a32fa706b97
$ git cat-file blob refs/data/MD5/477e602800c18624d9bc7a32fa706b97 | md5sum
477e602800c18624d9bc7a32fa706b97 -
### git gitlab-push ###
The `git gitlab-push` command is actually an alias for the
[git-gitlab-push](/Utilities/GitSetup/git-gitlab-push) script.
In addition to pushing the topic branch to Gerrit the script also detects
content links added or modified by the commits in the topic.
It reads the data object hashes from the content links and looks for
matching `refs/data/` entries in the local Git repository.
The script pushes the matching data objects to your VTK GitLab fork.
For example:
$ git gitlab-push --dry-run --no-topic
* refs/data/MD5/477e602800c18624d9bc7a32fa706b97:refs/data/MD5/477e602800c18624d9bc7a32fa706b97 [new branch]
Pushed refs/data/MD5/477e602800c18624d9bc7a32fa706b97 and removed local ref.
A GitLab webhook that triggers whenever a topic branch is pushed checks
for `refs/data/` in your VTK GitLab fork, fetches them, erases the refs
from your fork, and uploads them to a location that we
[tell ExternalData to search](/CMake/vtkExternalData.cmake) at build time.
### Publishing Data for an External Branch ###
The above [workflow](#workflow) works well for developers working on a
single machine to contribute changes directly to upstream VTK. When
working in an external branch of VTK, perhaps during a long-term topic
development effort, data objects need to be published separately.
The workflow for adding data to an external branch of VTK is the same
as the above through the [commit](#commit) step, but diverges at the
[push](#push) step because one will push to a separate repository.
Our ExternalData infrastructure intentionally hides the real data files
from Git so only the content links (`.md5` files) will be pushed.
The real data objects will still be left in the `.ExternalData/MD5`
directory at the top of the VTK source tree by the
[pre-commit](#pre-commit) hook.
The `.ExternalData` directory must be published somewhere visible to
other machines that want to use the data, such as on a web server.
Once that is done then other machines can be told where to look for
the data, e.g.
cmake ../VTK "-DExternalData_URL_TEMPLATES=https://username.github.io/VTK/ExternalData/%(algo)/%(hash)
In this example we assume the files are published on a [Github Pages][]
`gh-pages` branch in `username`'s fork of VTK.
Within the `gh-pages` branch the files are placed at
`ExternalData/MD5/$md5sum` where `$md5sum` is the MD5 hash of the content
(these are the same names they have in the `.ExternalData` directory in
the original source tree).
[Github Pages]: https://help.github.com/articles/creating-project-pages-manually
Develop VTK with Git
====================
This page documents how to develop VTK through [Git][].
See the [README](README.md) for more information.
[Git]: http://git-scm.com
Git is an extremely powerful version control tool that supports many
different "workflows" for individual development and collaboration.
Here we document procedures used by the VTK development community.
In the interest of simplicity and brevity we do *not* provide an
explanation of why we use this approach.
Setup
-----
Before you begin, perform initial setup:
1. Register [GitLab Access].
2. Follow the [download instructions](download.md#clone) to create a
local VTK clone:
$ git clone https://gitlab.kitware.com/vtk/vtk.git VTK
$ cd VTK
3. Run the [developer setup script][] to prepare your VTK work tree and
create Git command aliases used below:
$ ./Utilities/SetupForDevelopment.sh
4. (Optional but highly recommended.)
[Register](https://open.cdash.org/register.php) with the VTK project
on Kitware's CDash instance to better know how your code performs in
regression tests. After registering and signing in, click on
"All Dashboards" link in the upper left corner, scroll down and click
"Subscribe to this project" on the right of VTK.
[GitLab Access]: https://gitlab.kitware.com/users/sign_in
[developer setup script]: /Utilities/SetupForDevelopment.sh
Workflow
--------
VTK development uses a [branchy workflow][] based on topic branches.
Our collaboration workflow consists of three main steps:
1. Local Development:
* [Update](#update)
* [Create a Topic](#create-a-topic)
2. Code Review (requires [GitLab Access][]):
* [Share a Topic](#share-a-topic)
* [Create a Merge Request](#create-a-merge-request)
* [Review a Merge Request](#review-a-merge-request)
* [Revise a Topic](#revise-a-topic)
3. Integrate Changes:
* [Merge a Topic](#merge-a-topic) (requires permission in GitLab)
* [Delete a Topic](#delete-a-topic)
[branchy workflow]: http://public.kitware.com/Wiki/Git/Workflow/Topic
Update
------
1. Update your local `master` branch:
$ git checkout master
$ git pull
Create a Topic
--------------
All new work must be committed on topic branches.
Name topics like you might name functions: concise but precise.
A reader should have a general idea of the feature or fix to be developed given just the branch name.
1. To start a new topic branch:
$ git fetch origin
2. For new development, start the topic from `origin/master`:
$ git checkout -b my-topic origin/master
For release branch fixes, start the topic from `origin/release`, and
by convention use a topic name starting in `release-`:
$ git checkout -b release-my-topic origin/release
3. Edit files and create commits (repeat as needed):
$ edit file1 file2 file3
$ git add file1 file2 file3
$ git commit
Caveats:
* To add data follow [these instructions](data.md).
* If your change modifies the `Utilities/KWSys/vtksys` directory please
contribute directly to [KWSys][] instead.
* If your change modifies the `Utilities/MetaIO/vtkmetaio` directory please
contribute directly to [MetaIO][] instead.
[KWSys]: http://public.kitware.com/Wiki/KWSys/Git
[MetaIO]: https://github.com/Kitware/MetaIO
Share a Topic
-------------
When a topic is ready for review and possible inclusion, share it by pushing
to a fork of your repository in GitLab. Be sure you have registered and
signed in for [GitLab Access][] and created your fork by visiting the main
[VTK GitLab][] repository page and using the "Fork" button in the upper right.
[VTK GitLab]: https://gitlab.kitware.com/vtk/vtk
1. Checkout the topic if it is not your current branch:
$ git checkout my-topic
2. Check what commits will be pushed to your fork in GitLab:
$ git prepush
3. Push commits in your topic branch to your fork in GitLab:
$ git gitlab-push
Notes:
* If you are revising a previously pushed topic and have rewritten the
topic history, add `-f` or `--force` to overwrite the destination.
* If the topic adds data see [this note](data.md#push).
The output will include a link to the topic branch in your fork in GitLab
and a link to a page for creating a Merge Request.
Create a Merge Request
----------------------
(If you already created a merge request for a given topic and have reached
this step after revising it, skip to the [next step](#review-a-merge-request).)
Visit your fork in GitLab, browse to the "**Merge Requests**" link on the
left, and use the "**New Merge Request**" button in the upper right to
reach the URL printed at the end of the [previous step](#share-a-topic).
It should be of the form:
https://gitlab.kitware.com/<username>/vtk/merge_requests/new
Follow these steps:
1. In the "**Source branch**" box select the `<username>/vtk` repository
and the `my-topic` branch.
2. In the "**Target branch**" box select the `vtk/vtk` repository and
the `master` branch. It should be the default.
If your change is a fix for the `release` branch, you should still
select the `master` branch as the target because the change needs
to end up there too.
3. Use the "**Compare branches**" button to proceed to the next page.
4. Fill out the form and use the "**Submit merge request**" button to
create the merge request and visit its page.
5. In the "**Description**" field provide a high-level description
of the change the topic makes and any relevant information about
how to try it. Use `@username` syntax to draw attention of
specific developers. If your change is a fix for the `release`
branch, indicate this so that a maintainer knows it should be
merged to `release`.
Optionally use a fenced code block with type `message`, such as
```message
This topic adds a really cool feature.
```
to specify text to be included in the generated merge commit message
when the topic is [merged](#merge-a-topic). Do not use `@username`
syntax in this block as it will not be interpreted.
Review a Merge Request
----------------------
Add comments mentioning specific developers using `@username` syntax to
draw their attention and have the topic reviewed.
### Human Reviews ###
Developers may add comments providing feedback or to acknowledge their
approval. Reviewers may end their comments with trailing lines of one
of these forms:
Acked-by: me
Reported-by: @someone
Reviewed-by: Some One <someone@somewhere.org>
Signed-off-by: ...
Tested-by: ...
These trailers will be extracted during [merging](#merge-a-topic) and
included as trailing lines of the generated merge commit message.
References to `me` and `@username` will automatically be transformed
into a real name and email address according to the user's GitLab
account profile.
### Robot Reviews ###
The "Kitware Robot" automatically performs basic checks on the commits
and adds a comment acknowledging or rejecting the topic. This will be
repeated automatically whenever the topic is pushed to your fork again.
A re-check may be explicitly requested by adding a comment with the
trailing line:
Do: check
A topic cannot be [merged](#merge-a-topic) until the automatic review
succeeds.
Revise a Topic