Debugging and Catalyst Replay
#############################

To simplify the process of debugging in-situ pipelines, catalyst now
supports the serialization of conduit_nodes. During each API call,
users can write the ``params`` argument of each API call out to disk.
Then, using ``catalyst_replay``, the nodes will be read back in,
and each API call will be invoked again. This prevents users from
needing to re-run their simulation when debugging.


Serializing Nodes and Writing to Disk
=====================================

To use the ``catalyst_replay`` command, nodes must first be written to disk.
The steps to do this are simple:

1. Set the environment variable ``CATALYST_DATA_DUMP_DIRECTORY`` to the directory
   where the node data for each API invocation should be saved.

2. Invoke the **stub** implementation in your custom API implementation.

This will write the ``conduit_node`` passed into the API call out to
``CATALYST_DATA_DUMP_DIRECTORY``. The ``conduit_nodes`` are written out as
``.conduit_bin`` files. They will follow the general pattern
``<stage>_params.conduit_bin.<num_ranks>.<rank>``, where:

1. ``<stage>`` is one of ``initialize``, ``execute`` or ``finalize``.

2. ``<num_ranks>`` is the number of MPI ranks that the simulation was run with.

3. ``<rank>`` is the 0 based index of the rank used to generate this file.

Files for the ``execute`` stage will also include the invocation number,
since ``catalyst_execute`` can be called multiple times. For example,
``execute_invc0_params.conduit_bin.2.1`` would contain the ``params`` passed
into the 0th invocation of ``catalyst_execute``, which was called by 2nd of
two ranks (since rank indices are 0-indexed).

Replaying API Calls with `catalyst_replay`
==========================================

After the node data has been written out to disk, the ``catalyst_replay``
command can be used to read the node data back into memory and execute the
same API calls. Find the ``catalyst_replay`` executable in the
``RUNTIME_OUTPUT_DIRECTORY`` generated by CMake (this is usually ``bin/``).
Run ``catalyst_replay`` with the same number of MPI ranks as the simulation
used to generate the data, and pass the value of ``CATALYST_DATA_DUMP_DIRECTORY``
as a command-line argument. This invoke each API method with the corresponding node
data. For an example, see the ``examples/replay`` directory.
