% -*- latex -*-

\chapter{Distributed systems}
\label{chap:distributedsys}
\index{DistributedSystems|(}

\section{Introduction}

As an HPC visualization toolkit, VTK-m is designed to be executed in
distributed systems.  This is a key requirement for resource expensive
applications where a computational job can be partitioned into different tasks
which are then assigned to different processes located in potentially different
nodes. Those nodes as a composite will normally form a cluster of computers or
a supercomputer.  These computing tasks can communicate and coordinate with
each other by the use of a \textit{middle-ware} like library which in the case
of VTK-m corresponds to the DIY library. Furthermore, for both launching the
jobs and ultimately to communicate between tasks we delegate into MPI (Message
Passing Interface).

Despite of the fact that only some of the VTK-m filters can run out-of-the-box
as a distributed job, many other VTK-m components can run as a distributed job
by manually partitioning the computational problem data and assigning to each
of the tasks one of those partitions.

\section{DIY}
\label{sec:diy}
\index{diy|(}

DIY is a block-parallel library for implementing scalable algorithms that can
execute both \textit{in-core} and \textit{out-of-core}. The same program can be
executed with one or more threads per MPI process, seamlessly combining
distributed-memory message passing with shared-memory thread parallelism. The
abstraction enabling these capabilities is block parallelism: blocks and their
message queues are mapped onto processing elements, consisting of either MPI
processes or threads, and are migrated between memory and storage by the DIY
runtime. Complex communication patterns, including neighbor exchange, merge
reduction, swap reduction, and \textit{all-to-all} exchange are possible in DIY
both \textit{in-core} and \textit{out-of-core}. An example of an DIY
application which finds the maximum of the computed medians of various
\vtkmcont{ArrayHandle} is shown in the listings \ref{ex:DIYSetupComm} and
\ref{ex:DIYForeach}.

DIY provides \textit{out-of-the-box} serialization of common C++
\textit{stdlib} types such as \textcode{std::vector} and
\textcode{std::string}, VTK-m also provides serialization for common VTK-m data
types such as: \vtkmcont{ArrayHandle}, and \vtkmcont{DataSet}. This list is not 
exhaustive, VTK-m also provides DIY serialization to many other data types.
For custom data types the user can specify how to serialize and
deserialize the desired type by defining an additional template specialization
for \textcode{struct vtkmdiy::Serialization}. An example of this can be found in
the listing~\ref{ex:DIYSerialization}.

\vtkmlisting[ex:DIYSetupComm]{Communication setup of an example DIY
application, (its communication topology can be found in the
figure~\ref{fig:DIYApp})}{DIYSetupComm.cxx}

\vtkmlisting[ex:DIYForeach]{Example DIY application which finds the maximum of the medians of
different \textidentifier{ArrayHandle} (its communication topology can be found
in the figure~\ref{fig:DIYApp})}{DIYForeach.cxx}

\begin{figure}[h!]
  \centering
  \includegraphics[width=5.5in]{images/DIYApp}
  \caption{
    Communication topology of the example DIY application shown in the listings
  \ref{ex:DIYSetupComm} and \ref{ex:DIYMasterExchange}.}
  \label{fig:DIYApp}
\end{figure}

\vtkmlisting[ex:DIYSerialization]{Example DIY application which displays how to
serialize custom data types in DIY.}{DIYSerialization.cxx}

\section{GPU-aware MPI}
\label{sec:gpu-aware-mpi}

Modern HPC GPUs allow direct \textit{GPU-to-GPU} communication. This provides
GPUs with an efficient mechanism to directly send data stored in their device
memory to the target GPU device memory. This is a significant departure from
the traditional approach where the NIC is solely accessible from the CPU
constraining us to a costly GPU communication pattern consisting in first
copying the desired data from the device memory to host memory, transferring it
over the network, and then again copying the received data from host memory to
device memory.

Both major GPUs parallel platforms ROCM and CUDA provide an API which supports
direct \textit{GPU-to-GPU} communication. Nevertheless, to avoid vendor lock in
VTK-m does not directly use these APIs. Instead, VTK-m delegates on MPI which
implements an unified and standardized API for \textit{GPU-to-GPU}
communication. Consequently, VTK-m provides the user with the capability of
using direct \textit{GPU-to-GPU} communication during MPI (distributed)
executions of VTK-m applications.

VTK-m can autonomously determine if \textit{GPU-to-GPU} communication is
possible. Consequently, it does not provide a specific API to control this
type of communication. This decision on the type of communication is done at
each call to the free function
\vtkmcont{DIYMasterExchange}\ref{ex:DIYForeach.cxx:DIYMasterExchange} which
internally decorates the method \textcode{DIY::Master::exchange} so that it can
perform this \textit{GPU-to-GPU} communication if the situation allows it.
Currently this is only possible with AMD GPUs that supports this feature such
as the AMD MI250X which is used by both OLCF Crusher and OLCF Frontier.

This \textit{GPU-aware} MPI feature can be enabled with the flag
\textcode{VTKm\_ENABLE\_GPU\_MPI=ON}. Lastly, enabling this feature in target
supercomputers often requires additional setup which is dependent on the
particular system, please refer to the target system documentation for further
information.
