\chapter{Visualizing Large Models}
\label{chap:VisualizingLargeModels}

ParaView is used frequently at Sandia National Laboratories and other
institutions for visualizing data from large-scale simulations run on the
world's largest supercomputers including the examples shown here.

\begin{inlinefig}
  \begin{tabular}{cc}
    \includegraphics[width=0.431\linewidth]{images/Asteroid} &
    \includegraphics[width=0.469\linewidth]{images/PolarVortex} \\
    \parbox[t]{0.431\linewidth}{\footnotesize CTH shock physics simulation
      with over 1 billion cells of a 10 megaton explosion detonated at the
      center of the Golevka asteroid.} &
    \parbox[t]{0.469\linewidth}{\footnotesize SEAM Climate Modeling
      simulation with 1 billion cells modeling the breakdown of the polar
      vortex, a circumpolar jet that traps polar air at high latitudes.}
  \end{tabular}

  \begin{tabular}{cc}
    \includegraphics[width=0.495\linewidth]{images/LargeAMR} &
    \includegraphics[width=0.405\linewidth]{images/vpic} \\
    \parbox[t]{0.474\linewidth}{\footnotesize A CTH simulation that
      generates AMR data. ParaView has been used to visualize CTH
      simulation AMR data comprising billions of cells, 100's of thousands
      of blocks, and eleven levels of hierarchy (not shown).} &
    \parbox[t]{0.426\linewidth}{\footnotesize A VPIC simulation of magnetic
      reconnection with 3.3 billion structured cells. Image courtesy of
      Bill Daughton, Los Alamos National Laboratory.}
  \end{tabular}

  \begin{tabular}{cc}
    \includegraphics[width=0.383\linewidth]{images/Crossflow} &
    \includegraphics[width=0.517\linewidth]{images/WingWake}
  \end{tabular}
  \parbox{0.95\linewidth}{\footnotesize ParaView visualizations run in situ
    with large scale PHASTA simulations. On the left is a 3.3 billion
    tetrahedral mesh simulating the flow over a full wing where a synthetic
    jet issues an unsteady crossflow jet (run on 160 thousand MPI
    processes). On the right is a 1.3 billion element mesh simulating the
    wake of a deflected wing flap (run on 256 thousand MPI processes).
    Images courtesy of Michel Rasquin, Argonne National Laboratory.}
\end{inlinefig}

In this section we discuss visualizing large meshes like these using the
parallel visualization capabilities of ParaView.  This section has
some exercises, but is less
``hands-on'' than the previous section.  Primarily you will learn the conceptual
knowledge needed to perform large parallel visualization.
The exercises demonstrate the basic techniques needed to
run ParaView on parallel machines.

The most fundamental idea to grasp is that when run on a large
machine every node processes different regions of the entire
dataset simultaneously. Thus the workable data resolution is limited by
the aggregate memory space of the machine.
We now present the basic ParaView architecture and parallel algorithms and
demonstrate how to apply this knowledge.

\section{Parallel Visualization Algorithms}
\label{sec:pva}

We are fortunate in visualization in that many operations are straightoward to parallelize. The data we deal with is
contained in a mesh, which means the data is already broken into little
pieces by the cells.  We can do visualization on a distributed parallel
machine by first dividing the cells among the processes.  For
demonstrative purposes, consider this very simplified mesh.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleMesh}
\end{inlinefig}

Now let us say we want to perform visualizations on this mesh using three
processes.  We can divide the cells of the mesh as shown below with the
blue, yellow, and pink regions.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExamplePartitions}
\end{inlinefig}

Once partitioned, many visualization algorithms will work by simply
allowing each process to independently run the algorithm on its local
collection of cells.  For example, take clipping (which is demonstrated in
multiple exercises including \ref{ex:UsingMultipleViews}).  Let us say that
we define a clipping plane and give that same plane to each of the
processes.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleClip1}
\end{inlinefig}

Each process can independently clip its cells with this plane.  The end
result is the same as if we had done the clipping serially.  If we were to
bring the cells together (which we would never actually do for large data
for obvious reasons) we would see that the clipping operation took place
correctly.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleClip2}
\end{inlinefig}

Many, but not all operations are thus trivially parallizeable. Other
operations are straightforward to parallelize if ghost layers as
described in Section~\ref{sec:ghostlayers} are available. Other operations require
more extensive data sharing among nodes. In these filters ParaView resorts to
MPI to communicate amongst the nodes of the machine.

\section{Basic Parallel Rendering}

\index{rendering!parallel|(}

When performing parallel visualization, we are careful to ensure that the
data remains partitioned among all of the processes up to and including
the rendering processes.  ParaView uses a parallel rendering library called
\keyterm{IceT}.  IceT uses a \keyterm{sort-last} algorithm for parallel
rendering.  This parallel rendering algorithm has each process
independently render its partition of the geometry and then
\keyterm{composites} the partial images together to form the final image.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelRendering}
\end{inlinefig}

The preceding diagram is an oversimplification.  IceT contains multiple
parallel image compositing algorithms such as \keyterm{binary tree},
\keyterm{binary swap}, and \keyterm{radix-k} that efficiently divide work
among processes using multiple phases.

\begin{inlinefig}
  \includegraphics[width=\linewidth]{images/ParallelRenderingDetail}
\end{inlinefig}

The wonderful thing about sort-last parallel rendering is that its
efficiency is completely insensitive to the amount of data being rendered.
This makes it a very scalable algorithm and well suited to large data.
However, the parallel rendering overhead does increase linearly with the
number of pixels in the image. This can be a problem for example when
driving tile display walls with ParaView. Consequently, some of the rendering
parameters described later in this chapter deal with limiting image size.

Because there is an overhead associated with parallel rendering, ParaView
can also render serially and will do so automatically when the visible
data is small enough. When the visible meshes are smaller than
a user defined threshold preference, or when parallel rendering is
turned off or unavailable, the geometry is shipped to the display
node, which then rasterizes it locally.  Obviously, this should only happen when the data being
rendered is small or the rendering process will be overwhelmed.

\section{ParaView Architecture}

With this introduction to parallel visualization, it is useful to
know something about how ParaView is structured and how it
orchestrates the parallel tasks described above.

ParaView is designed as a three-tier client-server architecture.  The three
logical units of ParaView are as follows.

\index{ParaView Server}
\begin{description}
\item[Data Server] \index{data server} The unit responsible for data
  reading, filtering, and writing.  All of the pipeline objects seen in the
  pipeline browser are contained in the data server.  The data server can
  be parallel.
\item[Render Server] \index{render server}The unit responsible for
  rendering.  The render server can also be parallel, in which case built
  in parallel rendering is also enabled.
\item[Client] \index{client}The unit responsible for establishing
  visualization.  The client controls the object creation, execution, and
  destruction in the servers, but does not contain any of the data (thus
  allowing the servers to scale without bottlenecking on the client).  If
  there is a GUI, that is also in the client.  The client is always a
  serial application.
\end{description}

These logical units need not be physically separated.  Logical units are
often embedded in the same application, removing the need for any
communication between them.  There are three modes in which you can run
ParaView. Note that no matter what mode ParaView runs in
the user interface and scripting API that you learned in
Chapters~\ref{chap:BasicUsage} and ~\ref{chap:BatchPythonScripting}
undergoes very little change.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/RunModeStandalone}
\end{inlinefig}

The first mode, which you are already familiar with, is
\keyterm{standalone} mode.  In standalone mode, the client, data server,
and render server are all combined into a single serial application.  When
you run the \progname{paraview} application, you are automatically connected
to a \keyterm{builtin} server so that you are ready to use the full
features of ParaView.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/RunModeClientServer}
\end{inlinefig}

The second mode is \keyterm{client-server} mode.  In client-server mode,
you execute the \progname{pvserver} program on a parallel machine and
connect to it with the \progname{paraview} client application.  The
\progname{pvserver} program has both the data server and render server
embedded in it, so both data processing and rendering take place there.
The client and server are connected via a socket, which is assumed to be a
relatively slow mode of communication, so data transfer over this socket is
minimized.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/RunModeClientRenderDataServer}
\end{inlinefig}

The third mode is \keyterm{client--render-server--data-server} mode.  In this
mode, all three logical units are running in separate programs.  As before,
the client is connected to the render server via a single socket
connection.  The render server and data server are connected by many socket
connections, one for each process in the render server.  Data transfer over
the sockets is minimized.

Although the client-render server-data server mode is supported, we almost
never recommend using it.  The original intent of this mode was to take
advantage of heterogeneous environments where one might have a large,
powerful computational platform and a second smaller parallel machine with
graphics hardware in it.  However, in practice we find any benefit is
almost always outstripped by the time it takes to move geometry from the
data server to the render server.  If the computational platform is much
bigger than the graphics cluster, then use software rendering on the large
computational platform.  If the two platforms are about the same size just
perform all the computation on the graphics cluster.

\section{Accessing a Parallel ParaView Server}

Accessing a standalone installation of ParaView is trivial.  Download
and install a pre-compiled binary or get it from your package manager, and go.  To visualize
extreme scale results, you need to access an installation of ParaView,
built with the MPI components enabled, on a machine
with sufficient aggregate memory to hold the entire data set and
derived data products. Accessing a
parallel enabled installation of ParaView's server is
intrinsically harder than acessing a standalone version.

Recent binary distributions of ParaView provided by Kitware do include
MPI for Mac, Linux and optionally Windows. The Windows MPI enabled binaries
depend upon Microsoft's MPI, which must be installed
separately. We will not cover those in the rest of this tutorial, but
feel free to ask questions on the ParaView mailing list.

Note that the specific MPI library version bundled into the
binaries were chosen for widest possible compatibility.
For production use ParaView will be much more effective when compiled with an MPI
version that is tuned for your machine's networking fabric. Your system
administrators can help you decide what MPI version to use.

To compile ParaView on a parallel machine you or your system
administrators will need the following.

\begin{itemize}
\item CMake cross-platform build setup tool
  (\href{http://www.cmake.org}{www.cmake.org})
\item MPI
\item OpenGL, either using a GPU in on- (X11) or off- screen (EGL) modes,
  or in software via Mesa.
\item Python +NumPy +Matplotlib (all optional but strongly recommended)
\item Qt $\ge$ 4.7 (optional)
\end{itemize}

Compiling without one of the optional libraries means a feature will not be
available.  Compiling without Qt means that you will not have the GUI
application and compiling without Python means that you will not have
scripting available. NumPy is almost as important as Python itself,
and Matplotlib is helpful for numeric text and some types of views and
color lookup tables.

To compile ParaView, you first run CMake, which will allow you to set up
compilation parameters and point to libraries on your system.  This will
create the make files that you then use to build ParaView.  For more
details on building a ParaView server, see the ParaView Wiki.

{
  \footnotesize
  \href{http://www.paraview.org/Wiki/Setting_up_a_ParaView_Server#Compiling}{http://www.paraview.org/Wiki/Setting\_up\_a\_ParaView\_Server\#Compiling}
}

Compiling ParaView on exotic machines, for example HPC class machines that
require cross compilation and some cloud-based on demand
systems,  can be an arduous task. You are welcome to seek free advice
on the ParaView mailing list or contracted assistance from Kitware.

Fortunately there are a number of large scale systems on which ParaView has
already been installed.  If you have are fortunate enough to have an
account on one of these systems, you
shouldn't need to compile anything. The ParaView community maintains
an opt in list of these with pointers to system specific documentation on the ParaView
Wiki. We invite system maintainers to add to the list if they are
permitted to.

{
  \footnotesize
  \href{http://www.paraview.org/Wiki/ParaView/HPC_Installations}{http://www.paraview.org/Wiki/HPC\_Installations}
}

Running ParaView in parallel is also intrinsically more difficult than
running the standalone client.  It typically involves a number of steps
that change depending on the hardware you are running on: logging in to
remote computers, allocating parallel nodes, launching a parallel program,
and sometimes tunneling through firewalls to establish interactive
connections to the compute nodes. These steps
are discussed in more detail in the next two sections in which we
finally return from theory to practical exercises.

\section{Batch Processing}
\label{sec:BatchProcessing}

ParaView's Python scripted interface, introduced in
Chapter~\ref{chap:BatchPythonScripting} has the important qualities of
being runnable offline, with the human removed from the loop, and
being inherently reproducible.

On large scale systems there are two steps that you need to
master. The first is to run ParaView in parallel through
MPI. The second is to submit a job through the system's queuing
system. For both steps the syntax varies from machine to machine and you should consult your
system's documentation for the details.

We demonstrate how to run ParaView in parallel with the simplest case of running Kitware's binary in
parallel on the PC class system that you have at hand.

\begin{exercise}{Running a visualization script in parallel}
 \label{ex:RunParallelBatch}%
You will need three things, the \progname{mpiexec} program to spawn MPI parallel programs, the \progname{pvbatch}
 executable and a Python script.

As for the Python script, let us use an example in which each node draws
a portion of a simple polygonal sphere with a color map that varies
over the number of processes in the job.

\begin{python}
from paraview.simple import *
sphere = Sphere()
rep = Show()
ColorBy(rep, ("POINTS", "vtkProcessId"))
Render()
rep.RescaleTransferFunctionToDataRange(True)
Render()
WriteImage("parasphere.png")
\end{python}

Use an editor to type this script into a file called parasphere.py and save it
somewhere. Note this is a stripped down version of a script recorded from a
parallel ParaView session in which we created a
 \gui{Sources} \ra \gui{Sphere} and chose the \gui{vtkProcessId} array to color by.

As for the executables, let us use the ones that come with ParaView.
After installing Kitware's ParaView binaries, you will find \progname{mpiexec} and \progname{pvbatch} at:
\begin{itemize}
\item On Mac:
\begin{pythonpluscommands}
  /Applications/ParaView-\pvversion.app/Contents/MacOS/mpiexec
\end{pythonpluscommands}
 and
\begin{pythonpluscommands}
  /Applications/ParaView-\pvversion.app/Contents/bin/pvbatch
\end{pythonpluscommands}
\item
On Linux and assuming you have extracted the binary into /usr/local:
\begin{pythonpluscommands}
  /usr/local/lib/paraview-\pvversion/mpiexec
\end{pythonpluscommands}
and
\begin{python}
  /usr/local/bin/pvbatch
\end{python}
Note that you will need to add ParaView's lib directory
(/usr/local/lib/paraview-\pvversion) to your LD\_LIBRARY\_PATH to use the mpi that comes with ParaView.
\item On Windows and assuming you installed the ``MPI'' version of ParaView
as well as the MS-MPI software (available separately from Microsoft for free):
\begin{python}
  C:/Program Files/Microsoft MPI/Bin/mpiexec
\end{python}
and
\begin{pythonpluscommands}
  C:/Program Files/ParaView \pvversion/bin/pvbatch
\end{pythonpluscommands}
\end{itemize}

Now we run the script in parallel by issuing this from the command
line.

\begin{itemize}
\item On Mac:
\begin{pythonpluscommands}
/Applications/ParaView-\pvversion.app/Contents/MacOS/mpiexec -np 4 \textbackslash
  /Applications/ParaView-\pvversion.app/Contents/bin/pvbatch \textbackslash
  parasphere.py
\end{pythonpluscommands}
\item
On Linux:
\begin{pythonpluscommands}
/usr/local/lib/paraview-\pvversion/mpiexec -np 4 \textbackslash
  /usr/local/bin/pvbatch \textbackslash
  parasphere.py
\end{pythonpluscommands}
\item
On Windows:
\begin{pythonpluscommands}
mpiexec -np 4 ^
  "C:/Program Files/ParaView \pvversion/bin/pvbatch" ^
  parasphere.py
\end{pythonpluscommands}
\end{itemize}

ParaView will run momentarily and it may or may not briefly display a
window on your desktop. In any case you will now find a file named
parasphere.png that looks like the following:

\begin{inlinefig}
  \includegraphics[width=0.431\linewidth]{images/parasphere}
\end{inlinefig}

\end{exercise}

Submitting a job through your system's queuing system typically
involves issuing a command like the following on the command line.

\begin{python}
qsub -A <project name to charge to compute time to> \
 -N <number of nodes>  \
 -n <number of processors on each node> \
 mpiexec -np <N*n> \
   pvbatch <arguments for pvbatch> \
   script_to_execute.py <arguments for Python script>
\end{python}

The command reserves the requested number of compute nodes on the
machine and, at some time in the future when the nodes become ready
for your use, spawns an MPI job. The MPI job runs ParaView's Python
scripted server in parallel, telling it to process the supplied script.

There are many queuing systems and mpi implementations and
site specific policies. Thus it it impossible to provide an exact
syntax that will work on every system here. Ask your system
administrators for guidance. Once you have the syntax, you should be
able to run the sphere example above to determine if you have a
working system.

\section{Interactive Parallel Processing}
\label{sec:InteractiveParallelProcessing}

For day to day work with large datasets it is feasible to
do interactive visualization in parallel too. In this mode you can
dynamically inspect the data and modify the visualization pipeline
while you work with the ParaView GUI on a workstation far from from
the HPC resource where the data is being processed.

In interactive configurations the data processing and rendering
portions work in parallel like above but they are controled from
the ParaView GUI instead of a Python script.

Starting the server process is similar to the above, but instead of
using the \progname{pvbatch} executable we use the \progname{pvserver} executable. The
former is limited to taking commands from Python scripts, the later is
built to take commands from a remote ParaView GUI program.

Client-server connections are established through the \texttt{paraview}
client application.  You connect to servers and disconnect from servers
with the \connect and \disconnect buttons.  When ParaView starts, it
automatically connects to the builtin server.  It also connects to
builtin whenever it disconnects~\disconnect from a server.

When you hit the \connect button, ParaView presents you with a dialog box
containing a list of known servers you may connect to.  This list of
servers can be both site- and user-specific.

You can specify how to connect to a server either through the GUI by
pressing the \gui{Add Server} button or through an XML definition file.
There are several options for specifying server connections, but ultimately
you are giving ParaView a command to launch the server and a host to
connect to after it is launched.

Once more we will demonstrate using the Kitware binaries and then
outline the steps required on large scale systems where the syntax
varies widely.

\begin{exercise}{Interactive Parallel Visualization}
 \label{ex:RunParallelInteractive}%
The \progname{pvserver} executable can be found in the same directory as the
\progname{pvbatch} exectuable. Let us begin by running it in parallel.

\begin{itemize}
\item On Mac:
\begin{pythonpluscommands}
/Applications/ParaView-\pvversion.app/Contents/MacOS/mpiexec -np 4 \textbackslash
  /Applications/ParaView-\pvversion.app/Contents/bin/pvserver &
\end{pythonpluscommands}
\item
On Linux:
\begin{pythonpluscommands}
/usr/local/lib/paraview-\pvversion/mpiexec -np 4 \textbackslash
  /usr/local/bin/pvserver &
\end{pythonpluscommands}
\item
On Windows:
\begin{pythonpluscommands}
mpiexec -np 4 ^
  "C:/Program Files/ParaView \pvversion/bin/pvserver"
\end{pythonpluscommands}
\end{itemize}

Now start up the ParaView GUI and click the \connect button. This will
bring the \gui{Choose Server Configuration} dialog box where you can download or create server
connection paths, or connect to predefined paths.

\begin{inlinefig}
  \includegraphics[width=0.7\linewidth]{images/connect}
\end{inlinefig}

Now click the \gui{Add Server} button to create a new path, which we
will construct to connect to the waiting \progname{pvserver} running on your
local machine.

\begin{inlinefig}
  \includegraphics[width=0.6\linewidth]{images/serverconfig}
\end{inlinefig}

In the \gui{Edit Server Configuration Dialog}, change the \gui{Name} of the
connection path from ``unknown'' to ``my computer''.
In actual use one would enter in a nickname for and the IP address of the
remote machine we want to connect to. Now hit the
\gui{Configure} button.

On the resulting \gui{Edit Server Launch Configuration} dialog, leave the \gui{Startup Type} as \gui{Manual} since we
have already started a waiting \progname{pvserver}. In actual use one would
typically use \gui{Command} and in the dialog enter in a script that
starts up the server on the remote machine. Click \gui{Save} on the
dialog to finish defining the connection.

\begin{inlinefig}
  \includegraphics[width=0.6\linewidth]{images/launchconfig}
\end{inlinefig}

Now that we have defined a connection, it will show up under the
\gui{Choose Server Configuration} dialog whenever we hit \connect. Click on the name of the
connection, ``my computer'' in this case, and click \gui{Connect} to
establish the connection between the GUI and \progname{pvserver}.

Once connected, simply \gui{File} \ra \gui{Open} files on the remote system and
work with them as before. To complete the exercise, open or create a
data set, and choose the \gui{vtkProcessId} array to examine how it is
partioned across the distributed memory of the server.

\end{exercise}

In practice, the syntax of starting up a parallel server on
a remote machine and connecting to it will be more involved.
Typically the steps include ssh'ing to a remote machine's login node, submitting
a job that runs a script that requests some number of nodes and runs \progname{pvserver} under
MPI on them with \progname{pvserver} made to connect back to the waiting client (instead of
the reverse as above) over at least one ssh tunnel hops. System administrators
and other adventurous folk can find details online here:

{
  \footnotesize
  \href
{http://www.paraview.org/Wiki/Setting_up_a_ParaView_Server#Running_the_Server}
{http://www.paraview.org/Wiki/Setting\_up\_a\_ParaView\_Server\#Running\_the\_Server}
}

{
 \footnotesize
  \href
{http://www.paraview.org/Wiki/Reverse_connection_and_port_forwarding}
{http://www.paraview.org/Wiki/Reverse\_connection\_and\_port\_forwarding}
}

On the local machine you may need helper utiliites as well. XQuartz (for X11) on
Mac and Putty (for ssh) on Windows are useful for starting up
remote connections. These are required to complete the next exercise.

Once a connection is established the steps should
be saved in scripts and configuration files. System administrators can
publish these so that authorized users can simply download and use
them. Several of these configuration files are hosted on the ParaView
website where the ParaView client can download them from.

\begin{exercise}{Fetching and using Connections}
\label{ex:FetchingConnections}%
Once again, click on \connect to bring up the \gui{Choose
Server Configuration} dialog. This time, hit the \gui{Fetch Servers}
button. This will bring up the \gui{Fetch Server Configurations} dialog
box populated with a list of server definitions from the webite.
Choose one and click the \gui{Import Selected} button to download it
into your ParaView preferences. The new connection is now available just as the connection to ``my computer'' is.

To use the remote machine, one more click \connect. This
time choose the new machine instead of ``my machine'' and click
\gui{Connect}. Doing so will bring up a \gui{Connection Options for ...}
dialog box that lets where you enter in the parameters for your parallel
session.

\begin{inlinefig}
  \includegraphics[width=0.7\linewidth]{images/connectoptions}
\end{inlinefig}

The parameters you will want to set include your username on
the remote machine, the number of nodes and processes to reserve and
the amount of time you want to reserve them for. Each site may have
additional choices that are passed into the launch scripts on the
server. Once set, click \gui{OK}  to
try to establish a connection.

When you click \gui{OK} ParaView typically spawns a terminal window
(xterm on Mac and Linux, cmd.exe on Windows). Here you will have to
enter in your credentials to actually access the remote machine. Once
logged on, the script to reserve the nodes automatically runs and in
most cases the terminal session has a menu with options for looking at the remote system's queue. When your
job reaches the top of the queue, the ParaView \gui{3D View} window will return
and you can begin doing remote visualization, this time with the
capacity of the remote machine at your fingertips.

\end{exercise}

\section{Parallel Data Processing Practicalities}

\subsection{Keeping Track of Memory}

\index{memory inspector|(}

When working with very large models, it is important to keep track of
memory usage on the computer. One of the most common and frustrating
problems encountered with large models is running out of memory. This in
turn will lead to thrashing in the virtual memory system or an outright
program fault.

Sections \ref{sec:AvoidingDataExplosion} and \ref{sec:CullingData} provide
suggestions to reduce your memory usage. Even so, it is wise to keep an eye
on the memory available in your system. ParaView provides a tool called the
\keyterm{memory inspector} designed to do just that.

\begin{inlinefig}
  \includegraphics[width=\scw]{images/MemoryInspector}
\end{inlinefig}

To access the memory inspector, select in the menu bar \gui{View} \ra
\gui{Memory Inspector}. The memory inspector provides information for both
the client you are running on and any server you might be connected to. It
will tell you the total amount of memory used on the system and the amount
of memory ParaView is using. For servers containing multiple nodes,
information both for the conglomerate job and for each individual node are
given. Note that a memory issue in any single node can cause a problem for
the entire ParaView job.

\index{memory inspector|)}

\subsection{Ghost Levels}
\label{sec:ghostlayers}

Unfortunately, blindly running visualization algorithms on partitions of
cells does not always result in the correct answer.  As a simple example,
consider the \keyterm{external faces} algorithm.  The external faces
algorithm finds all cell faces that belong to only one cell, thereby
identifying the boundaries of the mesh. What happens when we run external
faces independently on our partitions?

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleExternalFaces1}
\end{inlinefig}

Oops.  We see that when all the processes ran the external faces algorithm
independently, many internal faces were incorrectly identified as being
external.  This happens where a cell in one partition has a neighbor in
another partition.  A process has no access to cells in other partitions,
so there is no way of knowing that these neighboring cells exist.

The solution employed by ParaView and other parallel visualization systems
is to use \keyterm{ghost cells} (sometimes also called
\keyterm{halo regions}).  Ghost cells are cells that are held in one
process but actually belong to another.  To use ghost cells, we first have
to identify all the neighboring cells in each partition.  We then copy
these neighboring cells to the partition and mark them as ghost cells, as
indicated with the gray colored cells in the following example.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleExternalFaces2}
\end{inlinefig}

When we run the external faces algorithm with the ghost cells, we see that
we are still incorrectly identifying some internal faces as external.
However, all of these misclassified faces are on ghost cells, and the faces
inherit the ghost status of the cell it came from.  ParaView then strips
off the ghost faces and we are left with the correct answer.

In this example we have shown one layer of ghost cells: only those cells
that are direct neighbors of the partition's cells.  ParaView also has the
ability to retrieve multiple layers of ghost cells, where each layer
contains the neighbors of the previous layer not already contained in a
lower ghost layer or the original data itself.  This is useful when we have
cascading filters that each require their own layer of ghost cells.  They
each request an additional layer of ghost cells from upstream, and then
remove a layer from the data before sending it downstream.

\subsection{Data Partitioning}

For the most part, ParaView leaves the task of breaking up or
partitioning the data to the simulation code that produced the
data. It is then the responsibility of ParaView's specific reader
module for the file format in question to work efficiently, typically
reading different files from different nodes independently but simultaneously. In ideal
cases the rest of the parallel pipeline also operates independently
and simultaneously.

Still, in the general case, since we are breaking up and distributing our data, it is prudent to
address the ramifications of how we partition the data.  The data
shown in Section~\ref{sec:pva} has a \keyterm{spatially coherent} partitioning.  That
is, all the cells of each partition are located in a compact region of
space.  There are other ways to partition data.  For example, you could
have a random partitioning.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleRandomPartition1}
\end{inlinefig}

Random partitioning has some nice features.  It is easy to create and is
friendly to load balancing.  However, a serious problem exists with respect
to ghost cells.

\begin{inlinefig}
  \includegraphics[scale=\bbscale]{images/ParallelExampleRandomPartition2}
\end{inlinefig}

In this example, we see that a single level of ghost cells nearly
replicates the entire data set on all processes.  We have thus removed any
advantage we had with parallel processing.  Because ghost cells are used so
frequently, random partitioning is not used in ParaView.


\subsection{D3 Filter}

The previous sections described the importance of load balancing and ghost
levels for parallel visualization.  This section describes how to achieve
that.

Load balancing and ghost cells are handled automatically by ParaView when
you are reading structured data (image data, rectilinear grid, and
structured grid).  The implicit topology makes it easy to break the data
into spatially coherent chunks and identify where neighboring cells are
located.

It is an entirely different matter when you are reading in unstructured
data (poly data and unstructured grid).  There is no implicit topology and
no neighborhood information available.  ParaView is at the mercy of how the
data was written to disk.  Thus, when you read in unstructured data there
is no guarantee about how well load balanced your data will be.  It is also
unlikely that the data will have ghost cells available, which means that
the output of some filters may be incorrect.

Fortunately, ParaView has a filter that will both balance your unstructured
data and create ghost cells.  This filter is called D3, which is short for
distributed data decomposition.  Using D3 is easy; simply attach the filter
(located in \gui{Filters} \ra \gui{Alphabetical} \ra \gui{D3}) to whatever
data you wish to repartition.

\begin{inlinefig}
  \includegraphics[height=.3\linewidth]{images/D3ExampleBefore}
  \includegraphics[height=.3\linewidth]{images/D3ExampleAfter}
\end{inlinefig}

The most common use case for D3 is to attach it directly to your
unstructured grid reader.  Regardless of how well load balanced the incoming
data might be, it is important to be able to retrieve ghost cells so that
subsequent filters will generate the correct data.  The example above shows
a cutaway of the extract surface filter on an unstructured grid.  On the
left we see that there are many faces improperly extracted because we are
missing ghost cells.  On the right the problem is fixed by first using the
D3 filter.

\subsection{Ghost Cells Generator Filter}

In many cases a better alternative to D3 filter is the \gui{Ghost Level
Generator}. This filter is more efficient than the
D3 filter because it makes the assumption that the input unstructured
grid data is already partitioned into spatially coherent regions. This
is generally a safe assumption as simulations benefit from coherency too.
Because of this, it concerns itself only with cells at and near the
external shell of the mesh, and does not consider or transfer the bulk of
the data at the interior.

\section{Advice}

\subsection{Matching Job Size to Data Size}

\emph{How many cores should I have in my ParaView server?}  This is a
common question with many important ramifications.  It is also an
enormously difficult question.  The answer depends on a wide variety of
factors including what hardware each processor has, how much data is being
processed, what type of data is being processed, what type of visualization
operations are being done, and your own patience.

Consequently, we have no hard answer.  We do however have several rules of thumb.

\textbf{If you are loading structured data} (image data, rectilinear grid,
structured grid), try to have a minimum of one core per 20 million
cells.  If you can spare the cores, one core for every 5 to 10
million cells is usually plenty.

\textbf{If you are loading unstructured data} (poly data, unstructured
grid), try to have a minimum of one core per 1 million cells.  If you
can spare the cores, one core for every 250 to 500 thousand cells
is usually plenty.

As stated before, these are just rules of thumb, not absolutes.  You should
always try to experiment to gage what your core to data size should
be.  And, of course, there will always be times when the data you want to
load will stretch the limit of the resources you have available.  When this
happens, you will want to make sure that you avoid data explosion and that
you cull your data quickly.


\subsection{Avoiding Data Explosion}
\label{sec:AvoidingDataExplosion}

The pipeline model that ParaView presents is very convenient for
exploratory visualization.  The loose coupling between components provides
a very flexible framework for building unique visualizations, and the
pipeline structure allows you to tweak parameters quickly and easily.

The downside of this coupling is that it can have a larger memory
footprint.  Each stage of this pipeline maintains its own copy of the data.
Whenever possible, ParaView performs \keyterm{shallow copies} of the data
so that different stages of the pipeline point to the same block of data in
memory.  However, any filter that creates new data or changes the values or
topology of the data must allocate new memory for the result.  If ParaView
is filtering a very large mesh, inappropriate use of filters can quickly
deplete all available memory.  Therefore, when visualizing large data sets,
it is important to understand the memory requirements of filters.

Please keep in mind that the following advice is intended \emph{only for
  when dealing with very large amounts of data and the remaining available
  memory is low}.  When you are not in danger of running out of memory,
ignore all of the following advice.

When dealing with structured data, it is absolutely important to know what
filters will change the data to unstructured.  Unstructured data has a much
higher memory footprint, per cell, than structured data because the
topology must be explicitly written out.  There are many filters in
ParaView that will change the topology in some way, and these filters will
write out the data as an unstructured grid, because that is the only data
set that will handle any type of topology that is generated.  The following
list of filters will write out a new unstructured topology in its output
that is roughly equivalent to the input.  These filters should \emph{never}
be used with structured data and should be used with caution on
unstructured data.

%TODO: there are surely more filters in each category now

\ifthenelse{\boolean{savetrees}}{\noindent\begin{minipage}{\linewidth}}{}
\begin{multicols}{2}
  \begin{itemize}
  \item \gui{Append Datasets}
  \item \gui{Append Geometry}
  \item \gui{Clean}
  \item \gui{Clean to Grid}
  \item \gui{Connectivity}
  \item \gui{D3}
  \item \gui{Delaunay 2D/3D}
  \item \gui{Extract Edges}
  \item \gui{Linear Extrusion}
  \item \gui{Loop Subdivision}
  \item \gui{Reflect}
  \item \gui{Rotational Extrusion}
  \item \gui{Shrink}
  \item \gui{Smooth}
  \item \gui{Subdivide}
  \item \gui{Tessellate}
  \item \gui{Tetrahedralize}
  \item \gui{Triangle Strips}
  \item \gui{Triangulate}
  \end{itemize}
\end{multicols}
\ifthenelse{\boolean{savetrees}}{\end{minipage}}{}

Technically, the \gui{Ribbon} and \gui{Tube} filters should fall into this
list.  However, as they only work on 1D cells in poly data, the input data
is usually small and of little concern.

This similar set of filters also output unstructured grids, but they also
tend to reduce some of this data.  Be aware though that this data reduction
is often smaller than the overhead of converting to unstructured data.
Also note that the reduction is often not well balanced.  It is possible
(often likely) that a single process may not lose any cells.  Thus, these
filters should be used with caution on unstructured data and extreme
caution on structured data.

\ifthenelse{\boolean{savetrees}}{\noindent\begin{minipage}{\linewidth}}{}
\begin{multicols}{2}
  \begin{itemize}
  \item \gui{Clip}~\clip
  \item \gui{Decimate}
  \item \gui{Extract Cells by Region}
  \item \gui{Extract Selection}~\extractSelection
  \item \gui{Quadric Clustering}
  \item \gui{Threshold}~\threshold
  \end{itemize}
\end{multicols}
\ifthenelse{\boolean{savetrees}}{\end{minipage}}{}

Similar to the items in the preceding list, \gui{Extract
  Subset}~\extractSubset performs data
reduction on a structured data set, but also outputs a structured data set.
So the warning about creating new data still applies, but you do not have
to worry about converting to an unstructured grid.

This next set of filters also outputs unstructured data, but it also
performs a reduction on the dimension of the data (for example 3D to 2D),
which results in a much smaller output.  Thus, these filters are usually
safe to use with unstructured data and require only mild caution with
structured data.

\ifthenelse{\boolean{savetrees}}{\noindent\begin{minipage}{\linewidth}}{}
\begin{multicols}{2}
  \begin{itemize}
  \item \gui{Cell Centers}
  \item \gui{Contour}~\contour
  \item \gui{Extract CTH Parts}
  \item \gui{Extract Surface}
  \item \gui{Feature Edges}
  \item \gui{Mask Points}
  \item \gui{Outline (curvilinear)}
  \item \gui{Slice}~\slice
  \item \gui{Stream Tracer}~\streamTracer
  \end{itemize}
\end{multicols}
\ifthenelse{\boolean{savetrees}}{\end{minipage}}{}

These filters do not change the connectivity of the data at all.  Instead,
they only add field arrays to the data.  All the existing data is shallow
copied.  These filters are usually safe to use on all data.

\ifthenelse{\boolean{savetrees}}{\noindent\begin{minipage}{\linewidth}}{}
\begin{multicols}{2}
  \begin{itemize}
  \item \gui{Block Scalars}
  \item \gui{Calculator}~\calculator
  \item \gui{Cell Data to Point Data}
  \item \gui{Curvature}
  \item \gui{Elevation}
  \item \gui{Generate Surface Normals}
  \item \gui{Gradient}
  \item \gui{Level Scalars}
  \item \gui{Median}
  \item \gui{Mesh Quality}
  \item \gui{Octree Depth Limit}
  \item \gui{Octree Depth Scalars}
  \item \gui{Point Data to Cell Data}
  \item \gui{Process Id Scalars}
  \item \gui{Python Calculator}
  \item \gui{Random Vectors}
  \item \gui{Resample with dataset}
  \item \gui{Surface Flow}
  \item \gui{Surface Vectors}
  \item \gui{Texture Map to...}
  \item \gui{Transform}
  \item \gui{Warp (scalar)}
  \item \gui{Warp (vector)}~\warp
  \end{itemize}
\end{multicols}
\ifthenelse{\boolean{savetrees}}{\end{minipage}}{}

This final set of filters are those that either add no data to the output
(all data of consequence is shallow copied) or the data they add is
generally independent of the size of the input.  These are almost always
safe to add under any circumstances (although they may take a lot of time).

\ifthenelse{\boolean{savetrees}}{\noindent\begin{minipage}{\linewidth}}{}
\begin{multicols}{2}
  \begin{itemize}
  \item \gui{Annotate Time}
  \item \gui{Append Attributes}
  \item \gui{Extract Block}
  \item \gui{Extract Level}~\extractGroup
  \item \gui{Glyph}~\glyph
  \item \gui{Group Datasets}~\group
  \item \gui{Histogram}~\histogram
  \item \gui{Integrate Variables}
  \item \gui{Normal Glyphs}
  \item \gui{Outline}
  \item \gui{Outline Corners}
  \item \gui{Plot Global Variables Over Time}
  \item \gui{Plot Over Line}~\plotOverLine
  \item \gui{Plot Selection Over Time}~\plotSelectionOverTime
  \item \gui{Probe Location}~\probe
  \item \gui{Temporal Shift Scale}
  \item \gui{Temporal Snap-to-Time-Steps}
  \item \gui{Temporal Statistics}
  \end{itemize}
\end{multicols}
\ifthenelse{\boolean{savetrees}}{\end{minipage}}{}

There are a few special case filters that do not fit well into any of the
previous classes.  Some of the filters, currently \gui{Temporal
  Interpolator} and \gui{Particle Tracer}, perform calculations based on
how data changes over time.  Thus, these filters may need to load data for
two or more instances of time, which can double or more the amount of data
needed in memory.  The \gui{Temporal Cache} filter will also hold data for
multiple instances of time.  Also keep in mind that some of the temporal
filters such as the temporal statistics and the filters that plot over time
may need to iteratively load all data from disk.  Thus, it may take an
impractically long amount of time even though it does not require any extra
memory.

The \gui{Programmable Filter}~\icon{pqProgrammableFilter24} is also a
special case that is impossible to classify.  Since this filter does
whatever it is programmed to do, it can fall into any one of these
categories.

\subsection{Culling Data}
\label{sec:CullingData}

When dealing with large data, it is clearly best to cull out data whenever
possible, and the earlier the better.  Most large data starts as 3D
geometry and the desired geometry is often a surface.  As surfaces usually
have a much smaller memory footprint than the volumes that they are derived
from, it is best to convert to a surface soon.  Once you do that, you can
apply other filters in relative safety.

A very common visualization operation is to extract isosurfaces from a
volume using the \gui{Contour}~\contour filter.  The \gui{Contour} filter
usually outputs geometry much smaller than its input.  Thus, the
\gui{Contour} filter should be applied early if it is to be used at all.
Be careful when setting up the parameters to the \gui{Contour} filter
because it still is possible for it to generate a lot of data.  This
obviously can happen if you specify many isosurface values.  High
frequencies such as noise around an isosurface value can also cause a
large, irregular surface to form.

Another way to peer inside of a volume is to perform a \gui{Slice}~\slice
on it.  The \gui{Slice}~\slice filter will intersect a volume with a plane
and allow you to see the data in the volume where the plane intersects.  If
you know the relative location of an interesting feature in your large data
set, slicing is a good way to view it.

If you have little \emph{a-priori} knowledge of your data and would like to
explore the data without paying the memory and processing time for the full
data set, you can use the \gui{Extract Subset}~\extractSubset filter to
subsample the data.  The subsampled data can be dramatically smaller than
the original data and should still be well load balanced.  Of course, be
aware that you may miss small features if the subsampling steps over them
and that once you find a feature you should go back and visualize it with
the full data set.

There are also several features that can pull out a subset of a volume:
\gui{Clip}~\clip, \gui{Threshold}~\threshold, \gui{Extract Selection}, and
\gui{Extract Subset}~\extractSubset can all extract cells based on some
criterion.  Be aware, however, that the extracted cells are almost never
well balanced; expect some processes to have no cells removed.  Also, all
of these filters with the exception of \gui{Extract Subset}~\extractSubset
will convert structured data types to unstructured grids.  Therefore, they
should not be used unless the extracted cells are of at least an order of
magnitude less than the source data.

When possible, replace the use of a filter that extracts 3D data with one
that will extract 2D surfaces.  For example, if you are interested in a
plane through the data, use the \gui{Slice}~\slice filter rather than the
\gui{Clip}~\clip filter.  If you are interested in knowing the location of
a region of cells containing a particular range of values, consider using
the \gui{Contour}~\contour filter to generate surfaces at the ends of the
range rather than extract all of the cells with the
\gui{Threshold}~\threshold filter.  Be aware that substituting filters can
have an effect on downstream filters.  For example, running the
\gui{Histogram}~\histogram filter after
\gui{Threshold}~\threshold will have an entirely different effect than
running it after the roughly equivalent \gui{Contour}~\contour filter.

\subsection{Downsamplng}

Downsampling is also fequently helpful for very large data.
For example, the \gui{Extract Subset} filter reduce the size of Structured
DataSets by simpling taking strided samples along the i, j and k axes.

For unstructured grids the \gui{Quadric Clustering} filter downsamples unstructured data
into a smaller dataset with similar appearance by averaging vertices
within cells of a grid. ParaView injects this algorithm into the
display pipeline while you interact with the camera as described in
the next section.

In some cases though, it can be more effective to work with structured data
than unstructured data. In particular structured data volume rendering algorithms
are usually much faster. In general structured data
representations are very compact, adding very little overhead to the
raw data values. Unstructured data types require 12 bytes of memory storage
for every every vertex and at least 8 bytes of storage to define each
cell beyond the normal storage for cell and point aligned
values. Structured types use a total of 36 bytes to represent the
entire set of vertices and cells in comparison.

Fortunately you can convert from unstructured to structured easily with the \gui{Resample
  To Image} filter.
When the sizes of the cells in the
input do not very widely, and can thus be approximated well by constant
sized voxels, this can be very effective.
 This filter internally makes use of the DIY2
block-parallel communication and computation library to communicate
and transfer data among the parallel nodes.

\begin{inlinefig}
  \includegraphics[height=0.6\linewidth]{images/ResampleToImage}
\end{inlinefig}

There is a companion filter \gui{Resample With DataSet} which takes
in a source and an input object and resamples values from the source onto
the input, which does not have to be a regular grid. This also uses
DIY2. This is used to assigning new  values onto a particular shaped object.

\section{Parallel Rendering Details}

\index{rendering|(}

Rendering is the process of synthesizing the images that you see based on
your data.  The ability to effectively interact with your data depends
highly on the speed of the rendering.  Thanks to advances in 3D hardware
acceleration, fueled by the computer gaming market, we have the ability to
render 3D quickly even on moderately priced computers.  But, of course, the
speed of rendering is proportional to the amount of data being rendered.
As data gets bigger, the rendering process naturally gets slower.

\index{rendering!interactive|see{interactive render}}
\index{rendering!still|see{still render}}

To ensure that your visualization session remains interactive, ParaView
supports two modes of rendering that are automatically flipped as
necessary.  In the first mode, \keyterm{still render}, the data is rendered
at the highest level of detail.  This rendering mode ensures that all of
the data is represented accurately.  In the second mode,
\keyterm{interactive render}, speed takes precedence over accuracy.  This
rendering mode endeavors to provide a quick rendering rate regardless of
data size.

While you are interacting with a 3D view, for example rotating, panning, or
zooming with the mouse, ParaView uses an interactive render.  This is
because during the interaction a high frame rate is necessary to make these
features usable and because each frame is immediately replaced with a new
rendering while the interaction is occurring so that fine details are less
important during this mode.  At any time when interaction of the 3D view is
not taking place, ParaView uses a still render so that the full detail of
the data is available as you study it.  As you drag your mouse in a 3D view
to move the data, you may see an approximate rendering while you are moving
the mouse, but the full detail will be presented as soon as you release the
mouse button.

The interactive render is a compromise between speed and accuracy.  As
such, many of the rendering parameters concern when and how lower levels of
detail are used.

\subsection{Basic Rendering Settings}
\label{sec:BasicRenderingSettings}

Some of the most important rendering options are the LOD parameters.
During interactive rendering, the geometry may be replaced with a lower
\keyterm{level of detail} (\keyterm{LOD}), an approximate geometry with
fewer polygons.

\begin{inlinefig}
  \includegraphics[width=.28\linewidth]{images/GeometricLODFull}
  \includegraphics[width=.28\linewidth]{images/GeometricLOD50}
  \includegraphics[width=.28\linewidth]{images/GeometricLOD10}
\end{inlinefig}

The resolution of the geometric approximation can be controlled. In the
proceeding images, the left image is the full resolution; the middle image
is the default decimation for interactive rendering, and the right image is
ParaView's maximum decimation setting.

The 3D rendering parameters are located in the settings dialog box which is
accessed in the menu from \gui{Edit} \ra \gui{Settings} (\gui{ParaView} \ra
\gui{Preferences} on the Mac).  The rendering options in the dialog
are in the \gui{Render View} tab.

\begin{inlinefig}
  \includegraphics[width=0.8\scw]{images/SettingsRendering}
\end{inlinefig}

The options pertaining to the geometric decimation for interactive
rendering are located in a section labeled \gui{Interactive Rendering
  Options}. Some of these options are considered advanced, so to access
them you have to either toggle on the advanced options with the
\icon{pqAdvanced26} button or search for the option using the edit box at
the top of the dialog. The interactive rendering options include the
following.

\begin{itemize}
\item \index{LOD Threshold} Set the data size at which to use a decimated
  geometry in interactive rendering. If the geometry size is under this
  threshold, ParaView always renders the full geometry. Increase this value
  if you have a decent graphics card that can handle larger data. Try
  decreasing this value if your interactive renders are too slow.
\item \index{LOD Resolution} Set the factor that controls how large the
  decimated geometry should be. This control is set to a value between 0
  and 1. 0 produces a very small number of triangles but possibly with a
  lot of distortion. 1 produces more detailed surfaces but with larger
  geometry. \icon{pqAdvanced26}
\item \index{interactive render!delay} Add a delay between an interactive
  render and a still render. ParaView usually performs a still render
  immediately after an interactive motion is finished (for example,
  releasing the mouse button after a rotation). This option can add a delay
  that can give you time to start a second interaction before the still
  render starts, which is helpful if the still render takes a long time to
  complete. \icon{pqAdvanced26}
\item \index{interactive render!outline} Use an outline in place of
  decimated geometry. The outline is an alternative for when the geometry
  decimation takes too long or still produces too much geometry. However, it
  is more difficult to interact with just an outline.
\end{itemize}

ParaView contains many more rendering settings. Here is a summary of some
other settings that can effect the rendering performance regardless of
whether ParaView is run in client-server mode or not. These options are
spread among several categories, and several are considered advanced.

\begin{description}
\item[\gui{Geometry Mapper Options}]~
  \begin{itemize}
  \item \index{immediate mode rendering} \index{display lists} Enable or
    disable the use of display lists. Display lists are internal structures
    built by graphics systems. They can potentially speed up rendering but
    can also take up memory.
  \end{itemize}
\item[\gui{Translucent Rendering Options}]~
  \begin{itemize}
  \item \index{depth peeling} Enable or disable depth peeling. Depth
    peeling is a technique ParaView uses to properly render translucent
    surfaces. With it, the top surface is rendered and then ``peeled away''
    so that the next lower surface can be rendered and so on.  If you find
    that making surfaces transparent really slows things down or renders
    completely incorrectly, then your graphics hardware may not be
    implementing the depth peeling extensions well; try shutting off depth
    peeling. \icon{pqAdvanced26}
  \item Set the maximum number of peels to use with depth peeling. Using
    more peels allows more depth complexity but allowing less peels runs
    faster. You can try adjusting this parameter if translucent geometry
    renders too slow or translucent images do not look correct.
    \icon{pqAdvanced26}
  \end{itemize}
\item[\gui{Miscellaneous}]~
  \begin{itemize}
  \item When creating very large datasets, default to the outline
    representation. Surface representations usually require ParaView to
    extract geometry of the surface, which takes time and memory. For data
    with size above this threshold, use the outline representation, which
    has very little overhead, by default instead.
  \item \index{rendering!performance} Show or hide annotation providing
    rendering performance information. This information is handy when
    diagnosing performance problems. \icon{pqAdvanced26}
  \end{itemize}
\end{description}

Note that this is not a complete list of ParaView rendering settings. We
have left out settings that do not significantly effect rendering
performance. We have also left out settings that are only valid for
parallel client-server rendering, which are discussed in
Section~\ref{sec:ParallelRenderParameters}.

\subsection{Image Level of Detail}

The overhead incurred by the parallel rendering algorithms is proportional
to the size of the images being generated.  Also, images generated on a
server must be transfered to the client, a cost that is also proportional
to the image size.  To help increase the frame rate during interaction,
ParaView introduces a new LOD parameter that controls the size of the
images.

During interaction while parallel rendering, ParaView can optionally
\index{subsample}subsample the image.  That is, ParaView will reduce the
resolution of the image in each dimension by a factor during interaction.
Reduced images will be rendered, composited, and transfered.  On the
client, the image is inflated to the size of the available space in the
GUI.

\begin{inlinefig}
  \includegraphics[width=.2\linewidth]{images/ImageLODFull}
  \includegraphics[width=.2\linewidth]{images/ImageLOD2}
  \includegraphics[width=.2\linewidth]{images/ImageLOD4}
  \includegraphics[width=.2\linewidth]{images/ImageLOD8}
\end{inlinefig}

The resolution of the reduced images is controlled by the factor with which
the dimensions are divided.  In the preceeding images, the left image has
the full resolution.  The following images were rendered with the
resolution reduced by a factor of 2, 4, and 8, respectively.

ParaView also has the ability to compress images before transferring them
from server to client. Compression, of course, reduces the amount of data
transferred and therefore makes the most of the available bandwidth.
However, the time it takes to compress and decompress the images adds to
the latency.

ParaView contains three different image compression algorithms for
client-server rendering. The oldest is a custom algorithm called
\keyterm{Squirt}, which stands for Sequential Unified Image Run Transfer.
Squirt is a run-length encoding compression that reduces color depth to
increase run lengths. The second algorithm uses the \keyterm{Zlib}
compression library, which implements a variation of the Lempel-Ziv
algorithm. Zlib typically provides better compression than Squirt, but
takes longer to perform and hence adds to the latency. The most recent
addition is the \keyterm{LZ4} algorithm which is tuned for fast compression
and decompression.

\subsection{Parallel Render Parameters}
\label{sec:ParallelRenderParameters}

\begin{inlinefig}
  \includegraphics[width=0.8\scw]{images/SettingsServer}
\end{inlinefig}

Like the other 3D rendering parameters, the parallel rendering parameters
are located in the settings dialog box, which is accessed in the menu from
\gui{Edit} \ra \gui{Settings} (\gui{ParaView} \ra \gui{Preferences} on the
Mac).  The parallel rendering options in the dialog are in the \gui{Render
  View} tab (intermixed with several other rendering options such as those
described in Section~\ref{sec:BasicRenderingSettings}). The parallel and
client-server options are divided among several categories, and several are
considered advanced.

\begin{description}
\item[\gui{Remote/Parallel Rendering Options}]~
  \begin{itemize}
  \item \index{remote render threshold} Set the data size at which to
    render remotely in parallel or to render locally. If the geometry is
    over this threshold (and ParaView is connected to a remote server), the
    data is rendered in parallel remotely and images are sent back to the
    client. If the geometry is under this threshold, the geometry is sent
    back to the client and images are rendered locally on the client.
  \item Set the sub-sampling factor for still (non-interactive) rendering.
    Some large displays have more resolution than is really necessary, so
    this sub-sampling reduces the resolution of all images displayed.
    \icon{pqAdvanced26}
  \end{itemize}
\item[\gui{Client/Server Rendering Options}]~
  \begin{itemize}
  \item \index{interactive render!subsample} \index{subsample} Set the
    interactive subsampling factor. The overhead of parallel rendering is
    proportional to the size of the images generated.  Thus, you can speed
    up interactive rendering by specifying an image subsampling rate.  When
    this box is checked, interactive renders will create smaller images,
    which are then magnified when displayed.  This parameter is only used
    during interactive renders. \icon{pqAdvanced26}
  \end{itemize}
\item[\gui{Image Compression}]~
  \begin{itemize}
  \item Before images are shipped from server to client, they optionally
    can be compressed using one of the three compression algorithms:
    Squirt\index{Squirt}, Zlib\index{Zlib} or LZ4\index{LZ4}.
    To make the compression more
    effective, each algorithm has one or more tunable parameters that
    let you customize the behavior and target level of compression. \icon{pqAdvanced26}
  \item Suggested image compression presets are provided for several common
    network types. When attempting to select the best image compression
    options, try starting with the presets that best match your connection.
    \icon{pqAdvanced26}
  \end{itemize}
\end{description}

\index{rendering!parallel|)}

\subsection{Parameters for Large Data}

The default rendering parameters are suitable for most users.  However,
when dealing with very large data, it can help to tweak the rendering
parameters.  The optimal parameters depend on your data and the hardware
ParaView is running on, but here are several pieces of advice that you
should follow.

\begin{itemize}
\item Try turning off display lists. Turning this option off will
  prevent the graphics system from building special rendering structures.
  If you have graphics hardware, these rendering structures are important
  for feeding the GPUs fast enough.  However, if you do not have GPUs,
  these rendering structures do not help much.
\item If there is a long pause before the first interactive render of a
  particular data set, it might be the creation of the decimated
  geometry. Try using an outline instead of decimated geometry for
  interaction. You could also try lowering the factor of the decimation to
  0 to create smaller geometry.
\item Avoid shipping large geometry back to the client. The remote
  rendering will use the power of entire server to render and ship images
  to the client.  If remote rendering is off, geometry is shipped back to
  the client.  When you have large data, it is always faster to ship images
  than to ship data (although if your network has a high latency, this
  could become problematic for interactive frame rates).
\item Adjust the interactive image sub-sampling for client-server rendering
  as needed.  If image compositing is slow, if the connection between
  client and server has low bandwidth, or if you are rendering very large
  images, then a higher subsample rate can greatly improve your interactive
  rendering performance.
\item Make sure \gui{Image Compression} is on.  It has a tremendous effect
  on desktop delivery performance, and the artifacts it introduces, which
  are only there during interactive rendering, are minimal.  Lower
  bandwidth connections can try using Zlib instead of Squirt compression.
  Zlib will create smaller images at the cost of longer
  compression/decompression times.
\item If the network connection has a high latency, adjust the parameters
  to avoid remote rendering during interaction. In this case, you can try
  turning up the remote rendering threshold a bit, and this is a place
  where using the outline for interactive rendering is effective.
\item If the still (non-interactive) render is slow, try turning on the
  delay between interactive and still rendering to avoid unnecessary
  renders.
\end{itemize}

\index{rendering|)}

\section{Catalyst}
\label{sec:Catalyst}

For small scale data, running the ParaView GUI in serial mode is
likely acceptable. For large scale data, interactive parallel
visualization is quite useful. For very large scale data, batch mode
parallel processing is more effective because of practical concerns
like batch system queue wait time before a parallel job is launched
and system policies that limit the size and duration of interactive jobs.

For extreme scale visualization even batch mode is beginning to be impractical though
as comparatively slow disks limit the size of data that can be saved off
by the simulation at runtime. In this domain ParaView's Catalyst configuration is
recommended as an \textit{in situ} analysis and visualization library.

\keyterm{Catalyst} is a visualization framework that
packages either portions or the entirety of ParaView's parallel server framework so that it can be
linked into and called from arbitrary simulation codes. Slow IO
systems are largely bypassed via an adaptor mechanism whereby the simulation code's data structures are
translated or ideally reused directly by ParaView while still in
RAM.

With Catalyst, full resolution simulation
products stay in memory instead of being saved to and later loaded
back from disk. As the simulation progresses, it periodically calls
into ParaView. In this way it is possible to immediately generate
smaller derived data sets, images, plots, etc. that would otherwise be generated during post processing after a
much longer, human in the loop, delay.

\begin{inlinefig}
  \includegraphics[width=1.5\scw]{images/catalyst}
\end{inlinefig}

As explained more fully in the ParaView Catalyst User's Guide there are two
components to making use of Catalyst. The first is to Catalyze the
simulation. This is where a software developer compiles Catalyst into the simulation, adds
a handful of (typically three) function calls to it, and fleshes out
an adaptor template with working code that produces VTK data sets
from the simulation's own internal data structures.

Once the simulation has been Catalyzed, day to day users need
to define what ParaView should do with the data it is given. Catalyst
can produce data extracts, images, statistical quantities, etc.
Any of these can be made with Python or even lower level Fortran, C or C++ coding to
maximize runtime efficiency, but it is more
flexible and convenient to do so with recorded Catalyst scripts
which are demonstrated in the next exercise.

With Catalyst scripts the simulation input deck references a Python file that defines the
visualization pipeline. The script can do nearly everything that ParaView
batch scripts do and is generally created with help from the ParaView GUI. To
create the script one loads a representative data set, creates a
pipeline, designates the set of inputs and outputs to and output from the
pipeline, and then simply saves out the Python file. Catalyst scripts are very
similar to recorded Python traces.

The user interface for definining the inputs and outputs is found
inside the the Catalyst Script Generator plugin. Our final exercise demonstrates
with a small example.

\begin{exercise}{Catalyst}
\label{ex:Catalyst}%
We demonstrate a Catalyst workflow using the ParaView binaries and
the PythonFullExample code from the ParaView source
tree. PythonFullExample is a toy simulation that uses numpy and mpi4py to update a regular grid as time
evolves. The example consists of four files. Open and read these to begin.

The computational kernel is in fedatastructures.py. Inspection of the
code shows that it has no particular dependency on ParaView, and
simply makes up a structured grid in parallel with numpy and mpi4py.

From fedatastructures.py:
\begin{python}
    def Update(self, time):
        self.Velocity = numpy.zeros((self.Grid.GetNumberOfPoints(), 3))
        self.Velocity = self.Velocity + time
        self.Pressure = numpy.zeros(self.Grid.GetNumberOfCells())
\end{python}

The main loop is found in the fedriver.py. If the doCoprocessing
variable is false, this falls back to a simple loop which runs the
simulation to updates the in-memory array over time. When the
variable is true the Catalyst library is exercised via three function
calls: initialize(), coprocess() and finalize().

From fedriver.py:
\begin{python}
import numpy
import sys
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

import fedatastructures

grid = fedatastructures.GridClass([10, 12, 10], [2, 2, 2])
attributes = fedatastructures.AttributesClass(grid)
doCoprocessing = True

if doCoprocessing:
    import coprocessor
    coprocessor.initialize()
    coprocessor.addscript("cpscript.py")

for i in range(100):
    attributes.Update(i)
    if doCoprocessing:
        import coprocessor
        coprocessor.coprocess(i, i, grid, attributes)

if doCoprocessing:
    import coprocessor
    coprocessor.finalize()
\end{python}

fedriver.py and fedatastructures.py represent the simulation code. The
Catalyst parts consist of the general purpose adaptor code, found
within in the coprocessor.py, and the pipeline definition, found in
cpscript.py.
In coprocessor.py note in
particular where the coprocess function
populates the dataDescription data structure to give fedatastructure's
content over to ParaView. The coprocessor also takes in, with the
addscript() call the pipeline definition file.

From coprocessing.py:
\begin{python}
    if coProcessor.RequestDataDescription(dataDescription):
        import fedatastructures
        imageData = vtk.vtkImageData()
        imageData.SetExtent(\
          grid.XStartPoint, grid.XEndPoint, \
          0, grid.NumberOfYPoints-1, \
          0, grid.NumberOfZPoints-1)
        imageData.SetSpacing(grid.Spacing)

        velocity = paraview.numpy_support.numpy_to_vtk(attributes.Velocity)
        velocity.SetName("velocity")
        imageData.GetPointData().AddArray(velocity)

        pressure = numpy_support.numpy_to_vtk(attributes.Pressure)
        pressure.SetName("pressure")
        imageData.GetCellData().AddArray(pressure)
\end{python}

The starting script for this example is called cpscript.py. This
simply saves out the input it is given in a format that the ParaView
GUI can readily understand. The internals of the PipelineClass is
essentially a ParaView batch script that takes as input not a file
but the datadescription class from the adaptor.

From cpscript.py:
\begin{python}
    class Pipeline:
      filename_6_pvti = \
          coprocessor.CreateProducer( datadescription, "input" )

      # create a new 'Parallel ImageData Writer'
      imageDataWriter1 = \
          servermanager.writers.XMLPImageDataWriter(Input=filename_6_pvti)

      # register the writer with coprocessor
      # and provide it with information such as the filename to use,
      # how frequently to write the data, etc.
      coprocessor.RegisterWriter(imageDataWriter1, \
          filename="fullgrid_%t.pvti", freq=100)

      Slice1 = Slice( \
          Input=filename_6_pvti, guiName="Slice1", \
          Crinkleslice=0, SliceOffsetValues=[0.0], \
          Triangulatetheslice=1, SliceType="Plane" )
      Slice1.SliceType.Offset = 0.0
      Slice1.SliceType.Origin = [9.0, 11.0, 9.0]
      Slice1.SliceType.Normal = [1.0, 0.0, 0.0]

      # create a new 'Parallel PolyData Writer'
      parallelPolyDataWriter1 = \
          servermanager.writers.XMLPPolyDataWriter(Input=Slice1)

      # register the writer with coprocessor
      # and provide it with information such as the filename to use,
      # how frequently to write the data, etc.
      coprocessor.RegisterWriter(\
          parallelPolyDataWriter1, filename='slice_%t.pvtp', freq=10)

    return Pipeline()
\end{python}

Now let us run the simulation code to produce some data.
\begin{itemize}
\item On Mac:
\begin{pythonpluscommands}
/Applications/ParaView-\pvversion.app/Contents/MacOS/mpiexec \textbackslash
  -np 2 \textbackslash
  /Applications/ParaView-\pvversion.app/Contents/bin/pvbatch  --symmetric \textbackslash
  fedriver.py
\end{pythonpluscommands}
\item On Linux:
\begin{pythonpluscommands}
/usr/local/lib/paraview-\pvversion/mpiexec -np 2 \textbackslash
  /usr/local/bin/pvbatch --symmetric \textbackslash
  fedriver.py
\end{pythonpluscommands}
\item On Windows:
\begin{pythonpluscommands}
mpiexec -np 2 ^
  "C:/Program Files/ParaView \pvversion/bin/pvbatch" --symmetric ^
  fedriver.py
\end{pythonpluscommands}
\end{itemize}

The example will run and produce several files, among them
fullgrid\_0.pvti which is a raw dump of the simulation's full
output. Let us use that to customize Catalyst's output.

First open the file in the ParaView GUI and inspect it. You will find
that this is a very simple structured grid with two variables,
\gui{pressure} and \gui{velocity}.
Now insert the \gui{File} \ra \gui{Extract Subset} filter into the pipeline to demonstrate some simple but useful processing.

Next let's use the \gui{Export Inspector} to declare a set of files that you want Catalyst to produce at run time.
As we will see in a moment, the ParaView GUI is able to generate Catalyst scripts by capturing the current state of the visualization
constructed in ParaView. However, the state does not include file creation,
which is typically done in interactive sessions via \gui{File} \ra \gui{Save Data} or \gui{File} \ra \gui{Save Screenshot} actions.

Since ParaView 5.6, the \gui{Export Inspector} is the place for defining Catalyst's data products.
Open it by clicking \gui{Export Inspector} under the \gui{Catalyst} menu.
By default the panel will show up in new tab near the \gui{Properties Panel}.
Select it, if necessary, to bring it to the front.

\begin{inlinefig}
  \includegraphics[width=0.6\linewidth]{images/exportinspector}
\end{inlinefig}

The \gui{Data Extracts} section at the top allows you to attach a writer to the currently active element within the visualization pipeline. Make sure \gui{ExtractSubset1} is selected in the \gui{Pipeline Browser} and then choose \gui{XMLPImageDataWriter} from the file format pull down to the right of this filter name in \gui{Data Extracts}. To confirm that you want Catalyst to perform this particular data export at run time, click on the checkbox to the right of the file format.

If you want to change the writer's default settings, click on the configure button with the \gui{...} label to the right of the checkbox. In the \gui{Save Data Options} dialog that pops up, you can controls things like the compression level, the frequency at which Catalyst generates files, and the filename. The default filename is composed from the producing filter's name, the time step and the file format extension. Let's change it from
``ExtractSubset1\_\%t.pvti'' to ``smallgrid\_\%t.pvti''.
Here ``\%t'' gets replaced with the iteration number when the simulation runs.
Later, when we run the simulation with this script, we will get subsampled results instead of the full grid.

To add rendered images to the simulation run, configure them with the \gui{Image Extracts} section of the \gui{Export Inspector}. Just like with data extracts, you configure outputs for the currently active view here. Do so by making sure that ``RenderView1'' is active and then hit the checkbox to enable png dumps of this view.

To save your visualization session in the form of a Catalyst script, click \gui{Catalyst} \ra \gui{Export Catalyst Script}. Note that all of your choices in the Export Inspector are kept with the rest of the ParaView state, so \gui{File} \ra \gui{Save State} can make it easy to modify your setup after the fact.

Finally quit ParaView and run the simulation again. The files produced
will now include png files for screen captures at each time step, and vti
files corresponding to the output of the Extract Subset filter at each
timestep. Simply repeat the process, adding filters and views for
example, to set up any type of data generator for the simulation.

\end{exercise}

Note this is but one interface to using Catalyst. If simulation users
are uncomfortable with ParaView and/or Python, it is entirely possible for simulation
developers to instrument the simulation code with predefined and
parameterized pipelines with or without resorting to Python. It is
also possible to create minimized Catalyst editions that consist of just a
small portion of the larger ParaView code base.

As is the case with ParaView itself, Catalyst is evolving rapidly. It
already has more advanced offshoots than described
here. One, is a live data capability in which one can connect a
ParaView client to a running Catalyzed simulation in order to check in
on its progress from time to time and do a limited amount of
computational steering and debugging. Another is Cinema, which is an
image based framework for deferred visualization of automatically
generated simulation results organized in a database. For information on these we refer the reader to the ParaView
mailing list and websites.

% Chapter Visualizing Large Models
