% -*- latex -*-

\chapter{Distributed systems}
\label{chap:distributedsys}
\index{DistributedSystems|(}

\section{Introduction}

As an HPC visualization toolkit, VTK-m is designed to be executed in
distributed systems.  This is a key requirement for resource expensive
applications where a computational job can be partitioned into different tasks
which are then assigned to different processes located in potentially different
nodes. Those nodes as a composite will normally form a cluster of computers or
a supercomputer.  These computing tasks can communicate and coordinate with
each other by the use of a \textit{middle-ware} like library which in the case
of VTK-m corresponds to the DIY library. Furthermore, for both launching the
jobs and ultimately to communicate between tasks we delegate into MPI (Message
Passing Interface).

Despite of the fact that only some of the VTK-m filters can run out-of-the-box
as a distributed job, many other VTK-m components can run as a distributed job
by manually partitioning the computational problem data and assigning to each
of the tasks one of those partitions.

\section{DIY}
\label{sec:diy}
\index{diy|(}

DIY is a block-parallel library for implementing scalable algorithms that can
execute both \textit{in-core} and \textit{out-of-core}. The same program can be
executed with one or more threads per MPI process, seamlessly combining
distributed-memory message passing with shared-memory thread parallelism. The
abstraction enabling these capabilities is block parallelism: blocks and their
message queues are mapped onto processing elements, consisting of either MPI
processes or threads, and are migrated between memory and storage by the DIY
runtime. Complex communication patterns, including neighbor exchange, merge
reduction, swap reduction, and \textit{all-to-all} exchange are possible in DIY
both \textit{in-core} and \textit{out-of-core}.

A full description of using DIY to perform distributed visualization is beyond the scope of this guide.
For a full description, reference the documentation provided by DIY.
The basic procedure of any DIY algorithm is to first define a set of blocks, assign them to the ranks of an MPI job, and define neighborhood relationships between them.
The following example demonstrates defining a set of blocks, one per MPI rank.

\vtkmlisting[ex:DIYSetupComm]{Communication setup of an example DIY application.}{DIYSetupComm.cxx}

\begin{didyouknow}
  When using DIY objects from inside \VTKm, use the objects in the mangled \textnamespace{vtkmdiy} rather than \textnamespace{diy}.
  \VTKm uses this mangled namespace to prevent conflicts if it is used with another library or executable that uses a different version of DIY.
\end{didyouknow}

Communication in DIY is managed by the \textcode{vtkmdiy::Master} object.
References to the defined blocks are added to the \textcode{Master} object.
You can then run an operation on each of these blocks using the \textcode{foreach} method, which is given a function to execute on each block.
This function is provided with a proxy that enables communicating data with other nodes using a variety of communication patterns.
These communications do not happen right away but rather are queued for later exchange.
This exchange is done by calling the free function \vtkmcont{DIYMasterExchange}.
The following example, which builds on the previous one, finds the median value of an array on each rank and then finds the maximum median value.
The blocks and communication used by these examples are outlined in Figure \ref{fig:DIYApp}.

\vtkmlisting[ex:DIYForeach]{Example DIY application which finds the maximum of the medians of
different \textidentifier{ArrayHandle}.}{DIYForeach.cxx}

\begin{figure}[t]
  \centering
  \includegraphics[width=5.5in]{images/DIYApp}
  \caption{
    Communication topology of the example DIY application shown in the listings
  \ref{ex:DIYSetupComm} and \ref{ex:DIYForeach}.}
  \label{fig:DIYApp}
\end{figure}

\begin{commonerrors}
  Normally, the DIY exchange process is done by calling the \textcode{Master::exchange} method.
  However, when using DIY with VTK-m, the exchange should be done instead by calling \vtkmcont{DIYMasterExchange}.
  This function allows \VTKm to enable state to interface between DIY and \VTKm data (without otherwise affecting exchanges that happen outside of \VTKm).
\end{commonerrors}

\section{Object Serialization}

When data are transferred among ranks, the format needs to be packed in a way the message layer understands.
Objects that might have complex structure typically need to be converted to one or more buffers through serialization.

DIY provides \textit{out-of-the-box} serialization of common C++
\textit{stdlib} types such as \textcode{std::vector} and
\textcode{std::string}, VTK-m also provides serialization for common VTK-m data
types such as \vtkmcont{ArrayHandle} and \vtkmcont{DataSet}. This list is not
exhaustive since VTK-m also provides DIY serialization to many other data types.
For custom data types the user can specify how to serialize and deserialize the
desired type by defining an additional template specialization for
\textcode{struct vtkmdiy::Serialization}. An example of this can be found in
the listing~\ref{ex:DIYSerialization}.

\vtkmlisting[ex:DIYSerialization]{Example DIY application which displays how to
serialize custom data types in DIY.}{DIYSerialization.cxx}

\section{GPU-aware MPI}
\label{sec:gpu-aware-mpi}

Modern HPC GPUs allow direct \textit{GPU-to-GPU} communication. This provides
GPUs with an efficient mechanism to directly send data stored in their device
memory to the target GPU device memory. This is a significant departure from
the traditional approach where the NIC is solely accessible from the CPU
constraining us to a costly GPU communication pattern consisting in first
copying the desired data from the device memory to host memory, transferring it
over the network, and then again copying the received data from host memory to
device memory.

Both major GPUs parallel platforms ROCM and CUDA provide an API which supports
direct \textit{GPU-to-GPU} communication. Nevertheless, to avoid vendor lock in
VTK-m does not directly use these APIs. Instead, VTK-m delegates on MPI which
implements an unified and standardized API for \textit{GPU-to-GPU}
communication. Consequently, VTK-m provides the user with the capability of
using direct \textit{GPU-to-GPU} communication during MPI (distributed)
executions of VTK-m applications.

VTK-m can autonomously determine if \textit{GPU-to-GPU} communication is
possible. Consequently, it does not provide a specific API to control this
type of communication. This decision on the type of communication is done at
each call to the free function
\vtkmcont{DIYMasterExchange} (demonstrated in Example \ref{ex:DIYForeach}, line \ref{ex:DIYForeach.cxx:DIYMasterExchange}), which
internally decorates the method \textcode{DIY::Master::exchange} so that it can
perform this \textit{GPU-to-GPU} communication if the situation allows it.
Currently this is only possible with AMD GPUs that supports this feature such
as the AMD MI250X which is used by both OLCF Crusher and OLCF Frontier.

This \textit{GPU-aware} MPI feature can be enabled with the flag
\textcode{VTKm\_ENABLE\_GPU\_MPI=ON}. Lastly, enabling this feature in target
supercomputers often requires additional setup which is dependent on the
particular system, please refer to the target system documentation for further
information.
