Summary:

- Step 1: Testing Kokkos itself using test_all_sandia

- Step 2: Testing of Kokkos integrated into Trilinos (scripts/trilinos-integration/*.sh)

- Step 3: Close all issues labeled "InDevelop"

- Step 4: Locally update CHANGELOG, merge into master, edit scripts/master_history.txt

- Step 5: Locally snapshot new master into corresponding Trilinos branch (develop or temporary), push with checking-test-sems.sh

- Step 6: Push local Kokkos master to GitHub (need Owner approval)

Steps 1, 2, and 4 include testing that may fail. These failures must be fixed either by pull requests to Kokkos develop, or by creating a new Trilinos branch for parts of Trilinos that must be updated. This is what usually takes the most time.


// -------------------------------------------------------------------------------- //


Step 1: The following should be repeated on enough machines to cover all
supported compilers. Those machines are:

    kokkos-dev
    ??? <- TODO: identify other machines

  1.1. Clone kokkos develop branch (or just switch to it)

         git clone -b develop git@github.com:kokkos/kokkos.git
         cd kokkos

  1.2. Create a testing directory

         mkdir testing
         cd testing

  1.3. Run the test_all_sandia script with no options to test all compilers

         nohup ../scripts/test_all_sandia &
         tail -f nohup.out                   # to watch progress

// -------------------------------------------------------------------------------- //

Step 2:
  2.1. Build and test Trilinos with 4 different configurations; Run scripts for white and shepard that are provided in kokkos/scripts/trilinos-integration. These scripts load their own modules/environment, so don't require preparation. You can run all four at the same time, use separate directories for each.

         mkdir serial
         cd serial
         nohup KOKKOS_PATH/scripts/trilinos-integration/shepard_jenkins_run_script_serial_intel &

  2.2. Compare the compile errors and test failures between updated and pristine versions. There may be compile failures that happen in both, tests that fail in both, and there may be tests that only fail some times (thus, rerun tests manually as needed).

// -------------------------------------------------------------------------------- //

Step 3: Close all issues labeled "InDevelop"
      
       Use the GitHub web interface: https://github.com/kokkos/kokkos/issues?q=is%3Aopen+is%3Aissue+label%3AInDevelop
       Select all with checkbox in upper left, "Mark as closed" 

// -------------------------------------------------------------------------------- //

Step 4: This step should be run on kokkos-dev

  4.1. If you don't have a GitHub token already, generate one for yourself (this will give you TOKEN):

       https://github.com/settings/tokens

  4.2. Get a clean copy of the Kokkos develop branch

       git clone -b develop git@github.com:kokkos/kokkos.git
       cd kokkos

  4.3. If you haven't already, install Ruby and the "github_changelog_generator" "gem"
       The github_changelog_generator is here: https://github.com/skywinder/github-changelog-generator
       Its compatible Ruby version can be found here: https://github.com/skywinder/github-changelog-generator/blob/master/.ruby-version
       Grab the corresponding Ruby version from here: https://www.ruby-lang.org/en/downloads/
       Follow the usual configure,make,make install process: https://www.ruby-lang.org/en/documentation/installation/#building-from-source
       Note that you will likely have to install to a non-default location with "./configure --prefix=/path"

  4.4. Generate the initial changelog. Use the most recent tag as OLDTAG (`git tag -l` can show you all tags). The NEWTAG is the new version number, e.g. "2.04.00". RUN THIS OUTSIDE THE KOKKOS SOURCE TREE!

       github_changelog_generator kokkos/kokkos --token TOKEN --no-pull-requests --include-labels 'InDevelop' --exclude-labels 'question,DevelopOnly' --enhancement-labels 'enhancement,Feature Request' --future-release 'NEWTAG' --between-tags 'NEWTAG,OLDTAG'
       cat CHANGELOG.md

       Note: you may need to set an HTTPS proxy in order for this script to work:
       export https_proxy=http://wwwproxy.sandia.gov:80

  4.5. Manually cleanup and commit the change log.
       (Copy the new section from the generated CHANGELOG.md to KOKKOS_PATH/CHANGELOG.md)
       (Make desired changes to CHANGELOG.md to enhance clarity (remove issues not noteworthy))
       (Commit the CHANGELOG.md locally to develop)

       The changelog commit message should be:

          Adding Changelog for Release A.B.CD

          Part of Kokkos C++ Performance Portability Programming EcoSystem A.B

  4.6. Merge develop into master. DO NOT FAST-FORWARD THE MERGE!!!!
       DO NOT USE AN OLD GIT VERSION!!!!

       (From kokkos directory):
       git checkout master
       git merge --no-ff develop

       The merge commit message should be:

          Merge branch 'develop' for A.B.CD

          Part of Kokkos C++ Performance Portability Programming EcoSystem A.B

  4.7. Update the tag in kokkos/scripts/master_history.txt

       Tag description: MajorNumber.MinorNumber.WeeksSinceMinorNumberUpdate
       Tag field widths: #.#.##
       date description: month:day:year
       date field widths: ##:##:####
       master description: SHA1 of previous master commit (use `git log --first-parent master`)
       develop description: SHA1 of merged develop branch (use `git log develop`)
       SHA1 field width: ######## (8 chars)

       # Append to scripts/master_history.txt:

       tag:  2.03.13    date: 07:27:2017    master: da314444    develop: 29ccb58a
       
       git commit --amend -a

       Keep the merge commit as described in 4.6

  4.8. Create the new tag:

       git tag -a #.#.##

         (type the following into the tag message (same as for step 4.7))
         tag: #.#.##
         date: mm/dd/yyyy
         master: sha1
         develop: sha1

  4.9. DO NOT PUSH YET !!!


// -------------------------------------------------------------------------------- //

Step 5: This step can be done on any SEMS machine (e.g. kokkos-dev). Actually, the checkin step requires lots of disk space and RAM. Use ceerws1113 if you have access to it.

  5.1. Clone the Trilinos corresponding branch (or just switch to it)

        git clone -b kokkos-develop git@github.com:trilinos/Trilinos.git
        TRILINOS_PATH=$PWD/Trilinos

  5.2. Snapshot Kokkos into Trilinos - this requires python/2.7.9 and that both Trilinos and Kokkos be clean - no untracked or modified files. Run the following outside of the Kokkos and Trilinos source trees.

        module load sems-python/2.7.9
        python $KOKKOS_PATH/scripts/snapshot.py $KOKKOS_PATH $TRILINOS_PATH/packages

       If snapshotting kokkos-kernels, use the snapshot.py in kokkos.

  5.3. Push this Trilinos branch to GitHub, open a pull request for it.
       The pull request title should be

       Kokkos + KokkosKernels Promotion To Version X.X.XX

       Mention @trilinos/kokkos and @trilinos/kokkos-kernels

       Add a short description of the most significant features and bug fixes.

       Then add:

       ## Kokkos Changelog

       And copy-paste the content for this release from the kokkos CHANGELOG.md file, then do the same for kokkos-kernels under:

       ## KokkosKernels Changelog

  5.4. Run checkin-test to push to trilinos using the CI build modules (gcc/4.9.3)

       cd $TRILINOS_PATH
       mkdir CHECKIN
       cd CHECKIN
       nohup ../cmake/std/sems/checkin-test-sems.sh --do-all --push &

       Although Trilinos has experimental Pull Request testing, it is not good enough to replace the checkin script yet.

  5.5. If there are failures, fix and backtrack. Otherwise, go to next step

// -------------------------------------------------------------------------------- //

Step 6: Push Kokkos master to GitHub (requires Owner permission).
      
       cd KOKKOS_PATH
       git push --follow-tags origin master 
