#!/bin/sh

# ----------------------------------------------------------------------------
#       Run the VisIt test suite
#
#  Each .py file in the tests directory contains tags indicating the class(es)
#  of tests the file belongs to. Additionally, each test can be run in one
#  of several modes. Classes are used to select WHICH tests to run. Modes are
#  used to decide HOW to run those tests.
#
#  CLASSES are identified in two places; in the test .py files
#  to identify which tests support belong to which classes and on the
#  command line to run-test to select which classes of tests to run.
#
#  MODES is used to control global modes in which tests are performed. The
#  current list of supported modes is "serial","parallel" and
#  "scalable". The runtest script accepts a
#  "-m(odes)" argument which identifies the mode(s) in which tests should be
#  performed. It is a comma separated list of modes. Note that it does
#  not make sense to pass a list of modes to runtest with mutually exclusive
#  modes. For example, it would not make sense to pass "serial,parallel".
#  To run both modes, runtest would have to be executed twice.
#
#  Finally, note that regardless
#  of the mode of a test, the resultant image SHOULD be identical to its
#  associated baseline. If we ever add a mode to do testing using hardware
#  rendering, this may not be the case due to minor differences in how the
#  hardware and software may render.
#
#  CLASSES is simple and arbitrary selection mechanism for selecting classes
#  of tests to run. Each test identifies in a comment block in the .py file
#  for the test the classes that it is. Then, the -classes argument to runtest
#  will find and run only those tests in the specified list of classes. The
#  initial list of classes is "nightly"
#
#  Finally, if you runtest as you always have, it will behave as it always
#  has. That is, it will execute all tests sequentially with no attempt at
#  scalable rendering or parallel.
#
#  Programmer: Jeremy Meredith
#  Date:       April 17, 2002
#
#  Modifications:
#    Jeremy Meredith, Thu Jul 18 23:50:04 PDT 2002
#    Moved tests into categorized subdirectories.  Made successful test
#    cases output as such without needing the -verbose flag.
#    Set the display to elysium for now, until full -nowin mode works.
#
#    Jeremy Meredith, Fri Jul 19 16:16:15 PDT 2002
#    Allowed specific files to be passed on the commandline.
#    Made the main page be links to subpages, one per test file.
#    Added a return code for small differences.
#
#    Jeremy Meredith, Thu Aug 29 15:11:01 PDT 2002
#    Added copying of log file to html directory.  Reversed order of 
#    past results.
#
#    Jeremy Meredith, Thu Nov 14 13:34:51 PST 2002
#    Added code to make sure permissions were set up correctly.
#
#    Jeremy Meredith, Fri Nov 15 12:54:51 PST 2002
#    Added a "summary" file.  Allowed running without the PIL library.
#
#    Jeremy Meredith, Mon Apr 14 09:57:21 PDT 2003
#    Propagate -verbose to the CLI.
#
#    Jeremy Meredith, Mon Jun  9 17:52:25 PDT 2003
#    Added "." to the python path.  This lets us pick up modules
#    from the test directory.
#
#    Mark C. Miller, 02Jul03
#    Added modes and classes functionality. Classes allows you to select
#    tests by class and modes allows you to select global operating modes.
#    Initial implementation supports "Scalable_Rendering" and "Parallel"
#
#    Jeremy Meredith, Mon Aug 18 15:19:23 PDT 2003
#    Added timings.
#
#    Mark C. Miller, 18Dec03
#    Removed use of modes as a selection criteria for tests.
#    Updated documentation in pre-amble to this shell script.
#
#    Mark C. Miller, 15Jan04
#    Modified html output to support multi-mode testing
#    Added a -skip command line option to specify a list of modes/tests to skip
#    Added a -limit command line option to specify per-test elapsed time limit
#
#    Mark C. Miller, Sat Jan 31 22:58:37 PST 2004
#    Added -dbg flag for debugging.  Modified 'post' functionality to
#    do highlighting for previous test history.
#
#    Jeremy Meredith, Mon Feb  2 12:40:57 PST 2004
#    Removed my display from the script.  Nowin mode is fully functional.
#
#    Mark C. Miller, Tue Mar  2 17:06:46 PST 2004
#    Added -i and -e command-line switches
#
#    Brad Whitlock, Tue Mar 30 09:37:24 PDT 2004
#    I added code to finish off the HTML page for a crashed test so we can
#    see how far it got before crashing. I also changed the HTML generation
#    here so tests that are skipped don't have links to HTML pages that did
#    not get generated due to a skipped test.
#
#    Mark C. Miller, Tue Mar 30 08:37:26 PST 2004
#    Added -dump option. Made most variables that represent CL arguments
#    be string valued and either empty string or specific value
#
#    Brad Whitlock, Tue Mar 30 17:13:26 PST 2004
#    Added -notrackmem option to turn off the nre memory usage tracking
#    features.
#
#    Brad Whitlock, Thu Apr 8 15:56:19 PST 2004
#    Added better support for deleting a lot of files.
#
#    Jeremy Meredith, Wed Apr 28 10:51:54 PDT 2004
#    Added check to make sure timings existed before trying to use them.
#
#    Jeremy Meredith, Thu Aug 12 10:17:57 PDT 2004
#    Added runtime in seconds to test case output.
#
#    Mark C. Miller, Mon Nov 22 17:42:22 PST 2004
#    Added -nonowin option
#
#    Mark C. Miller, Mon Nov 29 18:52:41 PST 2004
#    Changed how interactive and noPIL options are handled
#
#    Hank Childs, Thu Mar  3 09:22:05 PST 2005
#    Added dynamic load balancing (dlb) mode.
#
#    Mark C. Miller, Mon Mar 21 19:01:04 PST 2005
#    Made message about rebuilding test data conditional on verbose setting
#
#    Jeremy Meredith, Fri Apr  1 10:16:28 PST 2005
#    Made the non-purify post-to-host be sunburn instead of localhost.
#
#    Jeremy Meredith, Fri Apr  8 10:49:19 PDT 2005
#    Added -e to some echo commands.  Apparently this needs to be explicitly
#    set on linux (i.e. -E is the default) instead of the old behavior we
#    are used to from the suns.
#
#    Mark C. Miller, Thu Apr 21 09:37:41 PDT 2005
#    Changed much of how purify runs work. Now, purify is dealt with like
#    other modes. There is no longer a -purify option. Also, eliminated
#    purify baseline errorcounts and much of the html output for it
#    Added code to setup for a purify run.
#
#    Brad Whitlock, Tue Apr 26 10:54:51 PDT 2005
#    I changed the test for dlb so it does not get executed when it should
#    not be executed.
#
#    Mark C. Miller, Wed May 11 08:11:27 PDT 2005
#    Added args for minor/major error threshold. Added "Passed" message
#    for case when pass has non-zero error but is below threshold
#
#    Jeremy Meredith, Tue Jun  7 11:01:24 PDT 2005
#    Added a "-lessverbose", or "-lv", flag that prints normal status
#    without all the progress messages.
#
#    Hank Childs, Wed Aug 24 16:49:49 PDT 2005
#    Add support for test specific config files.
#
#    Jeremy Meredith, Wed Sep  7 12:05:23 PDT 2005
#    Enhanced grep for finding runtimes for mode-specific tests.
#
#    Mark C. Miller, Mon Jan 23 16:11:59 PST 2006
#    Made it so gzip'd purify results can be viewed from netscape even
#    if you don't "post" the results to the website
#
#    Mark C. Miller, Wed Aug 16 15:48:21 PDT 2006
#    Added logic to save off core files to cores dir
#
#    Mark C. Miller, Tue Sep  5 12:06:58 PDT 2006
#    Added code to quiet output from re-makes of data and purify targets
#    Added copy of runtimes.txt to visit's public_html dir
#
#    Mark C. Miller, Mon Nov 20 22:47:10 PST 2006
#    In regenerating the top-level index.html file, made find commands faster
#    by maxing the depth at 1 and grep commands faster by stopping after
#    first match.
#
#    Mark C. Miller, Tue Nov 28 23:13:08 PST 2006
#    Changed CL args for controlling error threshold. Changed text of 
#    some HTML from "Minor" and "Major" to "Acceptable" and "Unacceptable"
#    Removed DiffUsingCkSum stuff. Improved logic to use 'diffState' for
#    indicating state of difference result.
#
#    Mark C. Miller, Tue Nov 28 23:50:15 PST 2006
#    Changed maxdiff to meddiff
#
#    Mark C. Miller, Wed Nov 29 08:19:52 PST 2006 
#    Changed meddiff to avgdiff
#
#    Mark C. Miller, Sun Dec  3 12:20:11 PST 2006
#    Added -vargs option to support all CL args that are merely passed onto
#    visit as opposed to being interpreted here (or in Testing.py). This
#    resulted in removing -dump, -dbg options. Also added ability for
#    individual tests to set env. variables.
#
#    Mark C. Miller, Mon Mar 19 19:24:02 PDT 2007
#    Backgrounded the actual VisIt test jobs that get run. Added limitTest 
#    shell script to limit the amount of time any test runs. Needed to
#    re-direct output in the main loop in a bit of a funky way.
#
#    Mark C. Miller, Tue Mar 20 09:48:03 PDT 2007
#    Removed re-direction at end of 'for t in $testcases' loop. That was
#    causing the whole loop to get executed in a subshell. Removed use
#    of 'kill -9' on the limitTest script (see comment there). Set default
#    timeout to be 5 minutes (300 seconds). Removed artificially low
#    120 second time out used to confirm limitTest was working.
#
#    Mark C. Miller, Tue Mar 27 08:39:55 PDT 2007
#    Removed backgrounding of 'interactive' runs as well as limiting the time
#    for them to run. Removed setting of PYTHONPATH var to include
#    /usr/gapps/visit/python+/sunos_57/python2.1
#
#    Hank Childs, Thu May 17 10:23:50 PDT 2007
#    Make executable path be "../src/bin/visit", instead of "../bin/visit",
#    since that is the right path for the new Subversion repo.
#
#    Jeremy Meredith, Thu May 17 14:44:15 EDT 2007
#    I made the html, diff, and current directories be forcibly removed
#    and re-created.  (This was previously unsafe because they were
#    clearcase elements, but I have removed them from SVN control.)
#
#    Mark C. Miller, Mon Aug 27 18:00:06 PDT 2007
#    Change handling to test-specific enviornment to instead of creating
#    a file for each test, put the relevant strings into a shell var that
#    is expanded on the test line.
#
#    Mark C. Miller, Tue Aug 28 16:23:20 PDT 2007
#    Added ability for test to specify how to invoke VisIt to run itself.
#    Added setting of $nowin="" for interactive mode
#
#    Mark C. Miller, Mon Dec 17 16:06:48 PST 2007
#    Added ability to skip specific test cases in a .py file
#
#    Mark C. Miller, Tue Jan 29 14:59:48 PST 2008
#    Modified setting of PYTHONPATH to include 
#
#    Mark C. Miller, Wed Feb 20 21:18:29 PST 2008
#    Added logic to limit engine process ids
#
#    Mark C. Miller, Tue Mar  4 18:36:42 PST 2008
#    Reduced time to wait to get engine pids from 25 to 10 seconds; check
#    for existence of engine_pids.txt file before trying to read it.
#
#    Mark C. Miller, Fri May 16 08:39:57 PDT 2008
#    Fixed problems with detecting and saving cores.
#
#    Tom Fogal, Fri Jul 11 14:05:08 EDT 2008
#    I added a mode for IceT rendering.
#    
#    Mark C. Miller, Tue Aug  5 16:31:43 PDT 2008
#    Increased default time limit for a given .py file test from 300 to
#    600 seconds.
#
#    Mark C. Miller, Mon Dec  8 23:36:31 PST 2008
#    I added logic to provide useful error messages when it looks like the
#    test ran 'ok' but developer negected to call Exit() from .py file.
#
#    Cyrus Harrison, Tue Dec  8 09:59:42 PST 2009
#    Updated to reflect reorg of data directory.
#
#    Tom Fogal, Thu Mar 25 14:43:08 MDT 2010
#    Acceptable differences ("-pixdiff X") shouldn't count as a failure.
#
#    Mark C. Miller, Sun Mar 28 22:59:47 PDT 2010
#    Added a 'cifrendly' mode where if a test fails, it is run a second time
#    with verbose and debugging ON. Verbose output gets capture in ci logs
#    automatically and debug logs are cat'd so they appear there too.
#
#    Mark C. Miller, Mon Mar 29 08:40:44 PDT 2010
#    Made output of debug logs in cifriendly mode a little more efficient
#    and human friendly. Only those logs that contain indications of error
#    (e.g. 'SIG' or 'EXCEPT' or 'Except) are cat'd. When a log is cat'd, each
#    line is pre-pended with the test name and log name.
#
#    Mark C. Miller, Thu Apr  1 14:26:36 PDT 2010
#    Added ulimit command.
#
#    Eric Brugger, Thu Apr 22 12:46:21 PDT 2010
#    I modified the script so that the summary for each test will read
#    "Succeeded With Skips" or "Acceptable With Skips" when a partially
#    skipped test is successful and "Unacceptable" when a partially skipped
#    test has a non-skipped test fail.  Previously all these cases simply
#    had "Partially Skipped".
#
#    Eric Brugger, Fri Apr 23 14:38:05 PDT 2010
#    I modified the parsing of the modes command so that it only considers
#    the key words serial, parallel, and scalable in an arbitrary order.
#    Specifically, if it sees the keyword serial, it runs the serial test,
#    if it sees the keyword parallel in the absence of the keyword scalable,
#    it runs the parallel test, and if it sees the keywords parallel and
#    scalable, it runs the test scalable,parallel test.  This change allows
#    it to ignore any arbitrary host name.
#
#    Mark C. Miller, Wed May  5 08:28:46 PDT 2010
#    Removed some bash-specific coding I introduced in Mar 29, 2010 update.
#    Removed use of an array shell variable and 'let' statements.
#
#    Mark C. Miller, Fri Jul 16 10:18:12 PDT 2010
#    Allow just scalable or parallel to be parsed from VISIT_TEST_MODES
#
#    Mark C. Miller, Mon Nov  1 12:15:04 PDT 2010
#    Pass '-exec-timeout' and '-idle-timeout' instead of just '-timeout'
#
#    Cyrus Harrison, Fri Jan 13 13:31:07 PST 2012
#    Added hooks for beta report generation.
#
#    Cyrus Harrison, Fri Apr  6 17:49:29 PDT 2012
#    Use explicit call to python for beta report gen.
#
#    Eric Brugger, Wed Aug  1 11:59:16 PDT 2012
#    I corrected a bug where all the tests that were executed after a test
#    generated a core file were flagged as having failed.  This was caused
#    by the "file" command now returning the full path of the executable
#    instead of just the executable name, causing the "mv" command to fail.
#    Now the script just takes the portion of the name after the last "\".
#
#    Cyrus Harrison, Mon Aug 13 16:48:14 PDT 2012
#    Guts are now python, use visit_test_suite.py as main entry point.
#
#    Mark C. Miller, Tue Oct  9 10:42:24 PDT 2012
#    Surround $@ passed to python with double quotes ("). Otherwise,
#    something like '--vargs "-debug 5"' winds up getting passed as 3 sep.
#    words: --vargs, "-debug, 5"
# ----------------------------------------------------------------------------

# Call python test suite:
python visit_test_suite.py "$@"


