% -*- latex -*-


\chapter{Strategies}
\label{chap:Strategies}

\index{strategy|(}

\IceT contains several parallel image compositing algorithms.  The type of
compositing algorithm to use is selected by choosing a
\index{strategy}\keyterm{strategy}.  This chapter describes the underlying
algorithm of each strategy.  This user's guide will give qualitative
comparisons between the strategies, but for a more quantitative analysis,
see the following paper.

\begin{quote}
  Kenneth Moreland, Brian Wylie, and Constantine Pavlakos.  ``Sort-last
  parallel rendering for viewing extremely large data sets on tile
  displays,'' In \emph{Proceedings of IEEE Symposium on Parallel and
    Large-Data Visualization and Graphics}, October 2001, pp. 85--154.
\end{quote}

A strategy is specified using the \CFunc{icetStrategy} function.

\begin{Table}{3}
  \textC{void }\CFunc{icetStrategy}\textC{(}&\CType{IceTStrategy}&\CArg{strategy}\quad\textC{);}
\end{Table}

The \CArg{strategy} is set to one of the identifiers for the strategies
documented in the following sections of this chapter.  A string documenting
the current strategy can be retrieved with the \CFunc{icetGetStrategyName}
function.

To describe the \IceT compositing algorithms, we will use the example
parallel rendering problem shown in Figure~\ref{fig:ExampleInputs} where 6
processes are each rendering their own piece of a shuttle model to a two
tile display.

\begin{figure}
  \centering
  \includegraphics{images/AllInput}
  \includegraphics{images/CompositedInput}
  \caption[Example compositing problem.]{An example of six processes
    rendering to two tiles (top) and their composited image (bottom).}
  \label{fig:ExampleInputs}
\end{figure}

In this example, processes are denoted, in no particular order, by the
colors gray, red, blue, green, purple, and orange.  The colors of the
geometry correspond to the process that generated each piece.  Image
boarder colors denote the process that generates and holds that image.
(Apologies to those having troubles resolving the colors due to poor
display or vision deficiencies.  It should not be hard to follow the
descriptions either way.)

\section{Single Image Compositing}
\label{sec:Strategies:SingleImageCompositing}

\index{single~image~composite|(}
\index{compositing!single~image|(}

Before discussing the multi-tile image compositing algorithms implemented
by \IceT, we visit the standard single image compositing algorithms.  You
cannot directly select a single image compositing algorithm as a strategy
(most of the multi-tile algorithms work well in ``single-tile'' mode), but
these compositing algorithms are used as ``subroutines'' in some of the
multi-tile algorithms.  A reference to a
\index{single~image~composite~network}\keyterm{single image composite
  network} in the subsequent compositing algorithm descriptions refers to
the algorithms described here.

\subsection{Tree Compositing}

\index{compositing!tree|(}
\index{tree~composite|(}
\index{binary~tree~composite|(}

\IceT internally has two single image composite implementations.  The first
of which is the \keyterm{tree composite} algorithm (sometimes also called
binary tree composite due to its pair-wise grouping).  The basic network
for tree composite is shown in Figure~\ref{fig:BinaryTree}.

\begin{figure}
  \centering
  \includegraphics{images/BinaryTree}
  \caption[Tree composite network.]{Tree composite network.  Arrows
    represent the passing of data from one stage to the next.  Processes
    receiving multiple images composite them together.}
  \label{fig:BinaryTree}
\end{figure}

The tree compositing algorithm is organized in stages.  At each stage the
processes pair up.  One of the processes sends its data to its pair and
then drops out of the computation.  The receiving process combines the two
images (using the \index{compositing~operation}compositing operation
described in
Chapter~\ref{sec:Customizing_Compositing:Compositing_Operation}) and
continues to the next stage.  Processing continues until there is only one
image (and one process) remaining.

As just defined, the tree composite algorithm only handles process counts
that are a power of two (that is, the number of processes is equal to $2^i$
for some integer $i$).  \IceT handles non-powers of two gracefully.  At any
stage where the number of processes is not even and one of the processes
cannot be paired, that leftover process does nothing for that stage but
then continues to participate in the next stage.  An example of this can be
seen in the second stage of Figure~\ref{fig:BinaryTree}.

The advantages of tree composite are its regular and efficient data
transfers.  The limiting factor of tree compositing is that at each stage
of the algorithm half of the processes drop out of the computation.  Thus,
for more than a few processes tree compositing provides poor process
utilization.

\index{binary~tree~composite|)}
\index{tree~composite|)}
\index{compositing!tree|)}

\subsection{Binary-Swap Compositing}

\index{compositing!binary~swap|(}
\index{binary~swap~composite|(}

The second single image compositing algorithm used internally by \IceT is
the \keyterm{binary swap} algorithm.  The basic network for binary-swap
composite is shown in Figure~\ref{fig:BinarySwap}

\begin{figure}
  \centering
  \includegraphics{images/BinarySwap}
  \caption[Binary-swap composite network.]{Binary-swap composite network.
    Arrows represent the passing of data from one stage to the next.
    Processes receiving multiple images composite or stitch them together.
    At most stages each process divides its image data and distributes it.
    The distribution of image data can be inferred from the target images.}
  \label{fig:BinarySwap}
\end{figure}

Like tree compositing, binary swap is organized in stages, and at each
stage the processes pair up.  However, rather than have one process send
all the data to the other, the image space is divided in two and the
processes swap image data so that each process has all the data for part of
the image.  At the next stage, the processes pair up again, but with
different partners that have the same partition of the image.  Processing
continues until each of the $N$ processes have an image $1/N$ the size of
the original image.  At this point, all the processes send their sub-image
to the display processes where the images are stitched together.

As just defined, the binary-swap composite algorithm only handles processes
that are a power of two (that is, the number of processes is equal to $2^i$
for some integer $i$).  Some binary-swap implementations handle non-powers
of two by reducing the problem to the next largest power of two and
dropping the leftover processes, but \IceT handles non-powers of two more
gracefully than that.  Instead, \IceT first finds the largest group of
processes that is a power of two, makes a partition out of them, then finds
the next largest group of processes that remain that is a power of two,
makes a partition out of them, and so on.  Each partition runs binary-swap
independently up to the point where each process has its own piece of data.
At this point, the smaller partitions send their image data to processes of
the larger partitions, dividing up images where necessary.  The largest
partition then finishes the compositing in the normal way by collecting all
of the pieces.

An example of compositing with a non-power of two is given in
Figure~\ref{fig:BinarySwap}.  The six processes are partitioned first into
a group of 4 and then into a group of 2.  After swapping, the processes in
the smaller group send images to the larger group.  In this case, the purple
process sends image data to the gray and blue processes, and the orange
process sends to the red and green processes.

Like tree composite, binary swap exhibits regular and efficient data
transfers.  In addition, binary swap involves the use of all the processes
throughout most of the compositing.  Consequently, binary swap exhibits
very good process utilization and scaling with respect to the number of
processes on which it is run.

The most inefficient part of binary swap is the collection of image
fragments at the end, which is an extra step that tree composite does not
need to take.  Through some empirical studies, we found that the binary
tree algorithm was more efficient than binary swap on less then 8 processes and
less efficient on more than 8 processes.  Consequently, \IceT automatically
switches between the two algorithms based on the amount of processes
involved in the compositing.

\index{binary~swap~composite|)}
\index{compositing!binary~swap|)}

\subsection{Ordered Compositing}

\index{compositing!ordered|(}

In some applications, the order in which images are composited together
makes a difference (see the Volume Rendering section in
Chapter~\ref{sec:Customizing_Compositing:Volume_Rendering}).  The details
on how ordered compositing is achieved is not given here, but the basic
idea for both compositing algorithms is that they first swizzle the
processes so that their order matches the order in which the images need to
be composited together.  When compositing images together, they make sure
to maintain over/under constancy based on the swizzled ranks from the
originating processes.  The networks are also managed such that no two
images are composited that are not directly ``next'' to each other (that
is, there is no image that needs to be inserted between them).

\index{compositing!ordered|)}

\index{compositing!single~image|)}
\index{single~image~composite|)}

\section{Reduce Strategy}
\label{sec:Strategies:Reduce}

\index{strategy!reduce|(}
\index{reduce~strategy|see{strategy, reduce}}
\index{reduce~to~single~tile|see{strategy, reduce}}

An effective strategy implemented in \IceT is the \keyterm{reduce to single
  tile strategy} (or simply the reduce strategy).  In this strategy, the
multi-tile composite problem is efficiently reduced to a set of single
image compositing problems, which are well studied and discussed in the
previous section.  The reduce strategy is selected by calling
\CFunc{icetStrategy} with the \CEnum{ICET\_STRATEGY\_REDUCE} argument.

\begin{figure}
  \centering
  \includegraphics{images/ReduceComposite}
  \caption[Reduce strategy composite network.]{Composite network for reduce
    strategy.  Arrows represent the passing of data from one stage to the
    next.  Processes receiving multiple images composite them together.
    The single image composite network is described in a preceding
    section.}
  \label{fig:ReduceComposite}
\end{figure}

The reduce strategy is performed in two phases.  In the first phase,
processes are partitioned into groups, each of which is responsible for
compositing the image of one of the tiles.  The number of processes
assigned to each tile is proportional to the number of non-empty images
rendered for the corresponding tile.  In the example shown in
Figure~\ref{fig:ReduceComposite} there are a total of 9 non-empty images.
The left tile has 3 of the 9, that is $\frac{1}{3}$, of the images and thus
is assigned $\frac{1}{3} \times 6 = 2$ processes.  Likewise, the right
image is assigned $\frac{2}{3} \times 6 = 4$ processes.

When assigned processes to tiles, display processes and processes rendering
images to the tile are given preference.  In the example of
Figure~\ref{fig:ReduceComposite}, the gray and blue processes are assigned
to the left tile.  The remainder are assigned to the right tile.  Any image
generated by a process that does not belong to the destination tile is
transferred to a process assigned to the tile.  In the example, the three
processes that render two images, gray, red, and blue, each pass one of
their images to a process in the opposing process group.  All of these
transfers have unique senders and receivers and thus can happen
simultaneously.

In the second phase of the reduce strategy, each group of processes
independently composites its images together using one of the single image
compositing algorithms described in the preceding section.

The reduce strategy supports ordered compositing.  It does this by ensuring
in the first phase that processes receive only images that are ``near'' the
image they hold, that is, there is no other image in between the two images
in the visibility ordering.  The single image compositing algorithms of the
second phase each support their own ordered compositing.

\index{strategy!reduce|)}

\section{Split Strategy}
\label{sec:Strategies:Split}

\index{strategy!split|(}
\index{split~strategy|see{strategy, split}}
\index{tile~split~and~delegate|see{strategy, split}}

The \keyterm{tile split and delegate strategy} (or simply the split
strategy) is a simple algorithm that splits up tiles, assigns each piece to
a tile, and then sends image fragments directly to the tiles for
compositing.  The split strategy makes efficient use of processing
resources, but exhibits haphazard message passing which can cause issues on
some high speed interconnects.  The split strategy is selected by calling
\CFunc{icetStrategy} with the \CEnum{ICET\_STRATEGY\_SPLIT} argument.

\begin{figure}
  \centering
  \includegraphics{images/TileSplit}
  \caption[Split strategy composite network.]{Compositing for split
    strategy.  First tiles are split and assigned to processes (upper
    left).  Then each process simultaneously sends its images to the
    responsible process (upper right) and receives all sub-images for its
    piece (bottom).  The composited pieces are then collected and stitched
    together.}
  \label{fig:TileSplit}
\end{figure}

The split strategy first assigns processes to tiles similar to how they are
assigned in the reduce strategy described previously.  That is, the number
of processes per tile is proportional to the number of non-empty images
generated for it.  Each tile is then split up evenly amongst all processes
assigned to it.  In the example in Figure~\ref{fig:TileSplit}, the upper
left image shows that the left image is split between 2 processes and the
right image is split amongst 4 processes.

On being assigned a section of tile, each process prepares to receive data
from all the sending processes using asynchronous receives.  Each process
then renders its images, splits them up, and sends the sub-images to the
corresponding process.  When a process is ready and as it receives data,
the incoming images are composited together.  Once all of the incoming
images are composited, the complete sub-image is sent to the display process
to be stitched together.

The split strategy does not support ordered compositing.  Using the split
strategy in color blending mode will fail.

\index{strategy!split|)}

\section{Virtual Trees Strategy}
\label{sec:Strategies:Vertial_Trees}

\index{strategy!virtual~trees|(}
\index{virtual~trees|see{strategy, virtual trees}}

The \keyterm{virtual trees strategy} is based on the binary tree
compositing algorithm, but performs multiple composites simultaneously to
regain some of the load balance lost in the original algorithm.  The
virtual trees strategy has nice regular communications, but still suffers
from some load imbalance, particularly when using fewer tiles and in later
stages of the algorithm.  The virtual trees strategy is selected by calling
\CFunc{icetStrategy} with the \CEnum{ICET\_STRATEGY\_VTREE} argument.

\begin{figure}
  \centering
  \includegraphics{images/VTrees}
  \caption[Virtual trees composite network.]{Composite network for virtual
    trees strategy.  Arrows represent the passing of data from one state to
    the next.  Processes receiving multiple images composite them
    together.}
  \label{fig:VTreesComposite}
\end{figure}

The virtual trees strategy works by creating a ``virtual'' tree for each
tile.  Contained in each tree are processes that have rendered an image for
that display tile.  The algorithm proceeds much like the binary tree
composition algorithm except that the processes float amongst the trees,
helping with the compositing as they become available.
Figure~\ref{fig:VTreesComposite} shows an example of the virtual trees
compositing.  In particular, notice that the gray process takes part in the
left tree in stage 1, then floats to take part in the right tree in stage
2, and then returns to take part in the left tree in stage 3.

When necessary, the process must keep track of multiple images belonging to
different virtual trees.  Two conserve memory, images are not rendered
until they are needed.  Also, a process can only hold two images at a time:
one that it is sending and one that it is receiving.  If a process is
holding an image for one tile, it cannot receive an image for another tile
until it sends away the image it is holding.

The virtual trees strategy does not support ordered compositing.  Using the
virtual trees strategy in color blending mode will fail.

\index{strategy!virtual~trees|)}

\section{Serial Strategy}
\label{sec:Strategies:Serial}

\index{strategy!serial|(}
\index{serial~strategy|see{strategy, serial}}

Despite its name, the \keyterm{serial strategy} does not completely
serialize the image compositing process.  Rather, it serially addresses the
tiles, but performs parallel compositing for each tile.  The serial
strategy is selected by calling \CFunc{icetStrategy} with the
\CEnum{ICET\_STRATEGY\_SERIAL} argument.

\begin{figure}
  \centering
  \includegraphics{images/SerialComposite}
  \caption[Serial compositing network.]{Composite network for serial
    compositing.  One at a time, each tile is composited using a parallel
    single image composite network described in a previous section.}
  \label{fig:SerialComposite}
\end{figure}

The serial strategy iterates over all of the tiles.  For each tile, it
composites all the images for that tile using one of the single image
compositing algorithms described in that preceding section.  As
demonstrated in the example in Figure~\ref{fig:SerialComposite} images from
all processes are composited for each tile regardless of whether some of
them may be empty.

Since the single image compositing algorithms support ordered
compositing, the serial strategy also supports ordered compositing.

The serial strategy is really only implemented as a baseline algorithm to
compare other algorithms.  In general, the reduce strategy does at least as
well or outperforms the serial strategy.  We have observed this even in
single tile mode, possibly because the reduce strategy can throw away empty
images.

\index{strategy!serial|)}

\section{Direct Send Strategy}
\label{sec:Strategies:DirectSend}

\index{strategy!direct~send|(}
\index{direct~send~strategy|see{strategy, direct send}}

The \keyterm{direct send strategy} is the simplest of all the strategies.
Each process simply renders its images and sends them directly to the
display process where the images get composited, as shown in
Figure~\ref{fig:DirectSend}.  The direct send strategy is selected by
calling \CFunc{icetStrategy} with the \CEnum{ICET\_STRATEGY\_DIRECT}
argument.

\begin{figure}
  \centering
  \includegraphics{images/DirectSend}
  \caption[Direct send compositing network.]{Composite network for direct
    send compositing.  Arrows represent the passing of data from one
    process to another.  Receiving process composite all incoming images
    together.}
  \label{fig:DirectSend}
\end{figure}

The direct send strategy is usually a poor performer.  It was designed as a
low watermark to compare to other compositing strategies.  The direct send
strategy does, however, support ordered compositing.

\index{strategy!direct~send|)}

\section{Implementing New Strategies}

The \IceT API was written while its strategies were being developed.  As
such, the design yields for the relatively simplistic addition of new
strategies.  This section will provide the basic overview of how to add a
new strategy.  It is probably easiest to start by modifying your \IceT
source to insert your own strategy in the src/ice-t/strategies directory of
the \IceT source distribution.  It should be possible to create an external
library with a strategy in it, but there may be some complications with
finding include files and getting the proper exports.

A strategy in \IceT is created by simply defining an \CType{IceTStrategy}
object.  The \CType{IceTStrategy} type is generally meant to be an opaque
type so that most users to not have to be concerned with the strategy
innards.  However, the \IceT code base is currently quite stable and the
\CType{IceTStrategy} type rarely changes in any case.

The \CType{IceTStrategy} type is defined in the
\index{ice-t.h}\index{GL/ice-t.h}\textC{GL/ice-t.h} header file.
\begin{code}
typedef GLuint *IceTImage;
typedef struct _IceTStrategy {
    const char *name;
    GLboolean supports_ordering;
    IceTImage (*compose)(void);
} IceTStrategy;
\end{code}
The \textC{name} field in the \CType{IceTStrategy} type is a short string
identifying the strategy.  (It is the string returned by
\CFunc{icetGetStrategyName} when the strategy is active.)  The
\textC{supports\_ordering} field is a Boolean value indicating whether the
strategy supports ordered compositing.  Finally, and most importantly, the
\textC{compose} field is a function that is called during an invocation of
\CFunc{icetDrawFrame} to perform the rendering and compositing.  It takes
no arguments (the composition should rely on the current \IceT state), and
returns a composited image.  The \CType{IceTImage} type is opaque to
compositing algorithms, but several functions, discussed later in this
chapter, are available to create and manipulate them.

All it takes to make \IceT use a new strategy is to define and fill an
object of type \CType{IceTStrategy} and pass it to the \CFunc{icetStrategy}
function.  Typically this is done as a static object somewhere within your
library source.  For example, the declaration for the direct send strategy
looks like this.

\begin{code}
static IceTImage directCompose(void);

IceTStrategy ICET_STRATEGY_DIRECT = { "Direct", ICET_TRUE, directCompose };
\end{code}

The object is also exposed in the
\index{ice-t.h}\index{GL/ice-t.h}\textC{GL/ice-t.h} header file in a way
that does not require linking directly to the strategy function.
\begin{code}
ICET_STRATEGY_EXPORT extern IceTStrategy ICET_STRATEGY_DIRECT;
\end{code}

The \CEnum{ICET\_STRATEGY\_DIRECT} name is intentionally formed to look
like a C macro for an identifier, and it is intended to be used by the end
user as such.  (\CEnum{ICET\_STRATEGY\_EXPORT} is a real C macro that
performs library export magic.)  Your strategy should define a
\CType{IceTStrategy} in a similar manner.  The remainder of this section
describes how to implement the compose function.

\subsection{Internal State Variables for Compositing}

As stated previously, a strategy's compose function does not take any
arguments.  Instead, it gets all relevant information from the \IceT state.
Many of the relevant state variables are described in the documentation for
the \CFunc{icetGet} functions (as well as elsewhere throughout this
document).  There are also several ``hidden'' state variables for internal
use.  The ones specifically useful for within a composite function are
listed here (along with the variable type, number of entries, and a
description).  Note that these state variables generally should be read
from, not written to.

\begin{Description}[xxxxxxxx]
\item[\CEnum{ICET\_ALL\_CONTAINED\_TILES\_MASKS}] (boolean,
  \CEnum{ICET\_NUM\_TILES} $\times$ \CEnum{ICET\_NUM\_PROCESSES}) Contains
  an appended list of \CEnum{ICET\_CONTAINED\_TILES\_MASK} variables for
  all processes.  Given process $p$ and tile $t$, the entry at
  $p+\CEnum{ICET\_NUM\_TILES} \times t$ contains the flag describing
  whether process $p$ renders a non-blank image for tile $t$.  This
  variable is the same on all processes.
\item[\CEnum{ICET\_CONTAINED\_TILES\_LIST}] (integer,
  \CEnum{ICET\_NUM\_CONTAINED\_TILES}) All the tiles into which the local
  geometry projects.  In other words, this is the list of tiles which will
  not be empty after local rendering.  The local processor should generate
  images for these tiles and participate in the composition of them.
\item[\CEnum{ICET\_CONTAINED\_TILES\_MASK}] (boolean,
  \CEnum{ICET\_NUM\_TILES}) This is a list of boolean flags, one per tile.
  The flag is 1 if the local geometry projects onto the tile (that is, the
  local render will not be empty for that tile) and 0 otherwise.  This
  gives the same information as \CEnum{ICET\_CONTAINED\_TILES\_LIST}, but
  in a different way that can be more convenient in some circumstances.
\item[\CEnum{ICET\_CONTAINED\_VIEWPORT}] (integer, 4) Describes the region
  of the viewport that the geometry being rendered locally projects onto.
  The bounds of the data (given by \CFunc{icetBoundingBox} or
  \CFunc{icetBoundingVertices}) onto the tile display and determines the
  region of the tile display the data covers.  The values in the four-tuple
  correspond to x, y, width, and height, respectively, of the projection in
  global pixel coordinates.  This variable in conjunction with the
  \CEnum{ICET\_NEAR\_DEPTH} and \CEnum{ICET\_FAR\_DEPTH} give the full 3D
  projection of the local data in window space.
\item[\CEnum{ICET\_FAR\_DEPTH}] (double, 1) The maximum depth value of the
  local geometry after projection.  See \CEnum{ICET\_CONTAINED\_VIEWPORTS}
  for more details.
\item[\CEnum{ICET\_IS\_DRAWING\_FRAME}] (boolean, 1) Set to true while in a
  call to \CFunc{icetDrawFrame} and set to false otherwise.  This should
  always be set to true while the compose function is being executed.
\item[\CEnum{ICET\_NEAR\_DEPTH}] (double, 1) The minimum depth value of the
  local geometry after projection.  See \CEnum{ICET\_CONTAINED\_VIEWPORTS}
  for more details.
\item[\CEnum{ICET\_NUM\_CONTAINED\_TILES}] (integer, 1) The number of tiles
  into which the local geometry projects.  This is the length of the
  \CEnum{ICET\_CONTAINED\_TILES\_LIST} variable.
\item[\CEnum{ICET\_PROJECTION\_MATRIX}] (double, 16) The current projection
  matrix, read from OpenGL at the invocation of \CFunc{icetDrawFrame}.
\item[\CEnum{ICET\_TILE\_CONTRIB\_COUNTS}] (integer,
  \CEnum{ICET\_NUM\_TILES}) For each tile, provides the number of processes
  that will produce a non-empty image for that tile.
\item[\CEnum{ICET\_TOTAL\_IMAGE\_COUNT}] (integer, 1) The total number of
  non-empty images produced by all processes for all tiles.  This variable
  is the sub of all entries in \CEnum{ICET\_TILE\_CONTRIB\_COUNTS}.
\end{Description}

\label{manpage:icetUnsafeStateGet}
In addition to several internal state variables, \IceT also has several
internal functions for accessing them.  The most important one for
implementing a strategy is \CFunc{icetUnsafeStateGet}, which is defined in
the \index{state.h}\textC{state.h} header file.

\begin{Table}{3}
  \textC{void *}\CFunc{icetUnsafeStateGet}\textC{(}&\textC{GLenum}&\CArg{pname}\quad\textC{);}
\end{Table}

The implementation for the \CFunc{icetGet} functions is to copy the data
into a memory buffer you provide, performing type conversion as necessary.
\CFunc{icetUnsafeStateGet} simply returns the internal pointer where the
data is stored.  This can be faster and more convenient (since you do not
have to allocate your own memory), but is unsafe in two ways.  First, if
the state variable is changed, the pointer you receive can become invalid.
Second, no type conversion is performed.  You have to make sure you cast
the pointer correctly yourself, and there is no real way to query the
correct type.  Since the state setting functions are hidden from the end
user API, it is possible to manage these erroneous conditions.

\subsection{Memory Management}

Compositing algorithms by their nature require buffers of memory of
non-trivial size to hold images, among other data, that are not needed in
between calls to the compositing.  One approach is to simply use the
standard C \CFuncExternal{malloc} and \CFuncExternal{free} functions.
However, some implementations of
\CFuncExternal{malloc}/\CFuncExternal{free} are not always efficient, and
even the best implementations can have a tendency to fragment memory over
time as large buffers are allocated and released.

To ensure efficient memory allocation, \IceT provides its own memory
management that is simple but effective for its compositing operations.
\IceT keeps around a pool of memory to be used by various components of the
API.  To use the \index{memory~pool}\index{pool!memory}\keyterm{memory
  pool}, the code first clears the buffer and ensures that it is big
enough.  The code then reserves sections of the pool for various buffers.
Since this pool changes size infrequently, allocating time or memory
fragmentation is not an issue.  The size of the allocated memory is also
minimized since it is shared throughout all of \IceT.

\label{manpage:icetResizeBuffer}
The \CFunc{icetResizeBuffer} function is used to clear out the memory
pool.  The declaration for this function is located in the
\index{context.h}\textC{context.h} header file.

\begin{Table}{3}
  \textC{void }\CFunc{icetResizeBuffer}\textC{(}&\textC{int}&\CArg{size}\quad\textC{);}
\end{Table}

The \CFunc{icetResizeBuffer} function ensures that the memory pool is at
least \CArg{size} bytes large.  It also resets all previously allocated
memory (that is, freeing it back into the pool).  This has two important
consequences.  First, you must know the amount of memory you need
\emph{a-priori}.  You cannot resize the buffer once you have started
allocating memory blocks.  If you try to do so, the previously allocated
blocks (potentially) will be destroyed.

Second, since this block of data is shared amongst all functions of \IceT,
you must be aware that other \IceT code can potentially release your memory
and allocate its own.  You should feel free to use \IceT's memory pool from
within the compose function of your strategy and the image that it returns
is best allocated from this buffer.  Furthermore, the helper functions
described in this section to implement your own strategy are also safe to
call.  Be aware, however, that in between calls to your composite function
the memory you allocate will be lost and you will have to reallocate your
buffers.

\label{manpage:icetReserveBufferMem}
Once you have sized the memory pool, use \CFunc{icetReserveBufferMem} to
allocate a chunk of memory.

\begin{Table}{3}
  \textC{void *}\CFunc{icetReserveBufferMem}\textC{(}&\textC{int}&\CArg{size}\quad\textC{);}
\end{Table}

\CFunc{icetReserveBufferMem} returns a pointer to a buffer reserved to the
given size.  The buffer is aligned on 64-bit boundaries to help prevent
illegal memory accesses.

The following code snippet (taken from the direct send strategy)
demonstrates the use of \CFunc{icetResizeBuffer} and
\CFunc{icetReserveBufferMem}.  (The image size functions are described in
the section on image functions.)

\begin{code}
    icetResizeBuffer(  2*icetSparseImageSize(max_pixels)
                     + icetFullImageSize(max_pixels)
                     + num_tiles*sizeof(GLint));
    inImage     = icetReserveBufferMem(icetSparseImageSize(max_pixels));
    outImage    = icetReserveBufferMem(icetSparseImageSize(max_pixels));
    image       = icetReserveBufferMem(icetFullImageSize(max_pixels));
    tile_image_dest = icetReserveBufferMem(num_tiles*sizeof(GLint));
\end{code}

\subsection{Image Manipulation Functions}

You probably have noticed from the definition of the strategy structure
that the compose function returns a variable of type \CType{IceTImage}.
There is another variable type used internally by strategies called
\CType{IceTSparseImage}.  Both image types can hold color data or depth
data or both.  The \CType{IceTImage} type stores pixels as raw data, simple
2D arrays that are compatible with OpenGL buffers.  The
\CType{IceTSparseImage} stores images using
\index{active-pixel~encoding}active-pixel encoding, the run length encoding
described in the Active-Pixel Encoding section of
Chapter~\ref{sec:Customizing_Compositing:Active_Pixel_Encoding}.

Both the \CType{IceTImage} type and the \CType{IceTSparseImage} type are
opaque to compositing algorithms.  Although you will create them by
allocating a buffer and casting the pointer, you will not access the data
directly.  Instead, you will manipulate it with the functions described in
this section.  These functions are defined in the
\index{image.h}\textC{image.h} header file.

\subsubsection{Creating Images}

\label{manpage:icetFullImageSize}
\label{manpage:icetSparseImageSize}
To create an image, you first need to know how big of a buffer you need.
You can do this with the \CFunc{icetFullImageSize} and
\CFunc{icetSparseImageSize} functions.

\begin{Table}{3}
  \textC{GLuint }\CFunc{icetFullImageSize}\textC{(}&\textC{GLuint}&\CArg{pixels}\quad\textC{);}
\end{Table}
\begin{Table}{3}
  \textC{GLuint }\CFunc{icetSparseImageSize}\textC{(}&\textC{GLuint}&\CArg{pixels}\quad\textC{);}
\end{Table}

The former of these functions return the size, in bytes, required for an
\CType{IceTImage} containing the number of \CArg{pixels} specified.  The
latter performs the same operation for an \CType{IceTSparseImage}.  A
sparse image can vary in actual size depending on how well the data
compresses so \CFunc{icetSparseImageSize} conservatively returns the
maximum amount of bytes needed in any case.

To ensure memory is managed efficiently, your strategy will have to create
all of the images it uses by allocating them with \CFunc{icetResizeBuffer}
and \CFunc{icetReserveBufferMem} (discussed in the previous section with an
example) and then casting the pointer to \CType{IceTImage} or
\CType{IceTSparseImage} as appropriate.

\label{manpage:icetInitializeImage}
Image structures are basically a block of memory with a small bit of header
data.  \IceT functions that create images will fill that information for
you.  Occasionally you may need to explicitly fill the header information.
This is done with \CFunc{icetInitializeImage}.

\begin{Table}{3}
  \textC{void }\CFunc{icetInitializeImage}\textC{(}&\CType{IceTImage}&\CArg{image},\\
  &\textC{GLuint}&\CArg{pixel\_count}\quad\textC{);}
\end{Table}

\CFunc{icetInitializeImage} will initialize the \CArg{image} buffer to be
a full containing \CArg{pixel\_count} pixels and the type of pixel data
specified by the \CEnum{ICET\_INPUT\_BUFFERS} state parameter.  There are
only two common instances in which you will have to initialize an image
yourself.  The first is that you are filling the buffer one part at a
time.  The other is that you are creating a blank image, which frequently
happens when a tile is empty.  To clear out an image use
\CFunc{icetClearImage}.

\label{manpage:icetClearImage}
\begin{Table}{3}
  \textC{void }\CFunc{icetClearImage}\textC{(}&\CType{IceTImage}&\CArg{image}\quad\textC{);}
\end{Table}

\CFunc{icetClearImage} will set all of the pixel data in an image to the
background.  In practice, \CFunc{icetClearImage} is coupled with a call to
\CFunc{icetInitializeImage} such as in the following.

\begin{code}
  image = icetReserveBufferMem(icetFullImageSize(max_pixels));
  icetInitializeImage(image, max_pixels);
  icetClearImage(image);
\end{code}

\subsubsection{Querying Images}

\label{manpage:icetGetImagePixelCount}
Once you have an initialized image, whether initialized by you or some
other \IceT function, you can retrieve the number of pixels in it with
\CFunc{icetGetImagePixelCount}.

\begin{Table}{2}
  \textC{GLuint }\CFunc{icetGetImagePixelCount}\textC{(}&\CArg{image}\quad\textC{);}
\end{Table}

Unlike most functions, \CFunc{icetGetImagePixelCount} can take either a
\CType{IceTImage} or a \CType{IceTSparseImage}.

\label{manpage:icetGetImageColorBuffer}
\label{manpage:icetGetImageDepthBuffer}
If you need to access the actual data of a \CType{IceTImage}, you can do so
with \CFunc{icetGetImageColorBuffer} and \CFunc{icetGetImageDepthBuffer}.

\begin{Table}{4}
  \textC{GLubyte}&\textC{*}\CFunc{icetGetImageColorBuffer}\textC{(}&\CType{IceTImage}&\CArg{image}\quad\textC{);}\\
  \textC{GLuint}&\textC{*}\CFunc{icetGetImageDepthBuffer}\textC{(}&\CType{IceTImage}&\CArg{image}\quad\textC{);}
\end{Table}

For these functions to work, the image must be initialized and contain the
respective color or depth buffer (of course).  If this condition is not
met, an error is raised and \textC{NULL} is returned.

\subsubsection{Rendering Images}

\label{manpage:icetGetTileImage}
\label{manpage:icetGetCompressedTileImage}
To get the image for a particular tile in the display, use either
\CFunc{icetGetTileImage} or \CFunc{icetGetCompressedTileImage}.

\begin{Table}{3}
  \textC{void }\CFunc{icetGetTileImage}\textC{(}&\textC{GLint}&\CArg{tile},\\
    &\CType{IceTImage}&\CArg{buffer}\quad\textC{);}
\end{Table}
\begin{Table}{3}
  \textC{GLuint }\CFunc{icetGetCompressedTileImage}\textC{(}&\textC{GLint}&\CArg{tile},\\
    &\CType{IceTSparseImage}&\CArg{buffer}\quad\textC{);}
\end{Table}

Both functions will invoke a rendering for that tile (performing the
appropriate projection transformations) as necessary, read back the frame
buffers and store the results in an image buffer you specify.  The
difference, of course, is that \CFunc{icetGetTileImage} fills the buffer
with raw data whereas \CFunc{icetGetCompressedTileImage} will compress the
image data with \index{active-pixel~encoding}active-pixel encoding.

\CFunc{icetGetTileImage} writes a pre-determined amount of data into
\CArg{buffer}, which corresponds to the value returned by
\CFunc{icetFullImageSize}.  The amount of data written to \CArg{buffer} by
\CFunc{icetGetCompressedTileImage} varies depending on how well the image
compresses.  The actual number of bytes written is returned by
\CFunc{icetGetCompressedTileImage}.  In general, you should record this
size as you will need it to transfer the data to another process.  The
amount of data written will never exceed the amount returned by
\CFunc{icetSparseImageSize}.

\subsubsection{Compressing Images}

\label{manpage:icetCompressImage}
\CFunc{icetCompressImage} converts a full \CType{IceTImage} into to more
compact \CType{IceTSparseImage}.

\begin{Table}{3}
  \textC{GLuint }\CFunc{icetCompressImage}\textC{(}&\textC{const }\CType{IceTImage}&\CArg{imageBuffer},\\
  &\CType{IceTSparseImage}&\CArg{compressedBuffer}\textC{);}
\end{Table}

\CFunc{icetCompressImage} returns the actual size of
\CArg{compressedBuffer} in bytes.

\label{manpage:icetCompressSubImage}
Sometimes it is convenient to break up an image into pieces, and compress
each piece.  This is common when dividing up an image to be divided amongst
some amount of processes.  This can be most easily achieved by using the
\CFunc{icetCompressSubImage}.

\begin{Table}{3}
  \multicolumn{3}{l}{\textC{GLuint }\CFunc{icetCompressSubImage}\textC{(}}\\
  \makebox[2in]{}&\textC{const }\CType{IceTImage}&\CArg{imageBuffer},\\
  &\textC{GLuint}&\CArg{offset},\\
  &\textC{GLuint}&\CArg{pixels},\\
  &\CType{IceTSparseImage}&\CArg{compressedBuffer}\textC{);}
\end{Table}

\CFunc{icetCompressSubImage} compresses a region of contiguous pixels.  The
block of pixels starts \CArg{offset} pixels past the beginning of the image
and is \CArg{pixels} long.  \CArg{icetCompressImage} is equivalent to
calling \CArg{icetCompressSubImage} with \CArg{offset} set to $0$ and
\CArg{pixels} set to the result of \CFunc{icetGetImagePixelCount}.

\label{manpage:icetDecompressImage}
A sparse image can be returned to its uncompressed form with
\CFunc{icetDecompressImage}.

\begin{Table}{3}
  \multicolumn{3}{l}{\textC{GLuint }\CFunc{icetDecompressImage}\textC{(}}\\
  \makebox[2in]{}&\textC{const }\CType{IceTSparseImage}&\CArg{compressedBuffer},\\
  &\CType{IceTImage}&\CArg{imageBuffer}\quad\textC{);}
\end{Table}

\CFunc{icetDecompressImage} returns the number of pixels in the resulting
image, which is the same number that you will get if you call
\CFunc{icetGetImagePixelCount} on the resulting image.

\subsection{Communications}

The first thing to know about communications in \IceT is to understand that
it is up to the strategy to count how many bytes are being transmitted in
your compose function and store this in the \CEnum{ICET\_BYTES\_SENT} state
variable.  To make this easier, \index{common.h}\textC{common.h} (found in
the strategies directory) provides \CFunc{icetAddSentBytes}.

\label{manpage:icetAddSentBytes}
\begin{Table}{3}
  \textC{void }\CFunc{icetAddSentBytes}\textC{(}&\textC{GLint }&\CArg{num\_sending}\quad\textC{);}
\end{Table}

\CFunc{icetAddSentBytes} simply adds \CArg{num\_sending} to the value in
state variable \CEnum{ICET\_BYTES\_SENT}.  A call to
\CFunc{icetAddSentBytes} should be coupled with every communication call
that sends data.

\IceT provides an abstract communication layer, which is described in
detail in Chapter~\ref{chap:Communicators}.  A handle to a communicator is
stored in the current context.  To make using the communicator easier, a
set of convenience functions described next is available in the
\index{context.h}\textC{context.h} include file.  All of these functions
are based off of those found in the \MPI standard.  For documentation, see
that for the corresponding \MPI function.  Note that each function is
missing an argument specifying the communicator.  These functions just grab
the current context's communicator.

\index{ICET\_COMM\_DUPLICATE}
\renewcommand{\currentmansection}{ICET\_COMM\_DUPLICATE}
\begin{Table}{3}
  \textC{struct }\CType{IceTCommunicatorStruct}\textC{ *}\CFunc{ICET\_COMM\_DUPLICATE}\textC{(void);}
\end{Table}

\index{ICET\_COMM\_DESTROY}
\renewcommand{\currentmansection}{ICET\_COMM\_DESTROY}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_DESTROY}\textC{(void);}
\end{Table}

\index{ICET\_COMM\_SEND}
\renewcommand{\currentmansection}{ICET\_COMM\_SEND}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_SEND}\textC{(}&\textC{const void *}&\CArg{buf}\textC{,}\\
  &\textC{int}&\CArg{count},\\
  &\textC{GLenum}&\CArg{datatype},\\
  &\textC{int}&\CArg{dest},\\
  &\textC{int}&\CArg{tag}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_RECV}
\renewcommand{\currentmansection}{ICET\_COMM\_RECV}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_RECV}\textC{(}&\textC{void *}&\CArg{buf}\textC{,}\\
  &\textC{int}&\CArg{count},\\
  &\textC{GLenum}&\CArg{datatype},\\
  &\textC{int}&\CArg{src},\\
  &\textC{int}&\CArg{tag}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_SENDRECV}
\renewcommand{\currentmansection}{ICET\_COMM\_SENDRECV}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_SENDRECV}\textC{(}&\textC{const void *}&\CArg{sendbuf}\textC{,}\\
  &\textC{int}&\CArg{sendcount}\textC{,}\\
  &\textC{GLenum}&\CArg{sendtype}\textC{,}\\
  &\textC{int}&\CArg{dest}\textC{,}\\
  &\textC{int}&\CArg{sendtag}\textC{,}\\
  &\textC{void *}&\CArg{recvbuf}\textC{,}\\
  &\textC{int}&\CArg{recvcount}\textC{,}\\
  &\textC{GLenum}&\CArg{recvtype}\textC{,}\\
  &\textC{int}&\CArg{src}\textC{,}\\
  &\textC{int}&\CArg{recvtag}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_ALLGATHER}
\renewcommand{\currentmansection}{ICET\_COMM\_ALLGATHER}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_ALLGATHER}\textC{(}&\textC{const void *}&\CArg{sendbuf}\textC{,}\\
  &\textC{int}&\CArg{sendcount}\textC{,}\\
  &\textC{GLenum}&\CArg{type}\textC{,}\\
  &\textC{void *}&\CArg{recvbuf}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_ISEND}
\renewcommand{\currentmansection}{ICET\_COMM\_ISEND}
\begin{Table}{3}
  \CType{IceTCommRequest}\textC{ }\CFunc{ICET\_COMM\_ISEND}\textC{(}&\textC{const void *}&\CArg{buf}\textC{,}\\
  &\textC{int}&\CArg{count},\\
  &\textC{GLenum}&\CArg{datatype},\\
  &\textC{int}&\CArg{dest},\\
  &\textC{int}&\CArg{tag}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_IRECV}
\renewcommand{\currentmansection}{ICET\_COMM\_IRECV}
\begin{Table}{3}
  \CType{IceTCommRequest}\textC{ }\CFunc{ICET\_COMM\_IRECV}\textC{(}&\textC{void *}&\CArg{buf}\textC{,}\\
  &\textC{int}&\CArg{count},\\
  &\textC{GLenum}&\CArg{datatype},\\
  &\textC{int}&\CArg{src},\\
  &\textC{int}&\CArg{tag}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_WAIT}
\renewcommand{\currentmansection}{ICET\_COMM\_WAIT}
\begin{Table}{3}
  \textC{void }\CFunc{ICET\_COMM\_WAIT}\textC{(}&\CType{IceTCommRequest}\textC{ *}&\CArg{request}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_WAITANY}
\renewcommand{\currentmansection}{ICET\_COMM\_WAITANY}
\begin{Table}{3}
  \multicolumn{3}{l}{\textC{void }\CFunc{ICET\_COMM\_WAITANY}\textC{(}}\\
  \makebox[1.8in]{}&\textC{int}&\CArg{count},\\
  &\CType{IceTCommRequest}\textC{ *}&\CArg{array\_of\_requests}\quad\textC{);}
\end{Table}

\index{ICET\_COMM\_SIZE}
\renewcommand{\currentmansection}{ICET\_COMM\_SIZE}
\begin{Table}{3}
  \textC{int }\CFunc{ICET\_COMM\_SIZE}\textC{(void);}
\end{Table}

\index{ICET\_COMM\_RANK}
\renewcommand{\currentmansection}{ICET\_COMM\_RANK}
\begin{Table}{3}
  \textC{int }\CFunc{ICET\_COMM\_RANK}\textC{(void);}
\end{Table}

In each of these functions, the type parameter is set to one of the
following: \CEnum{ICET\_BOOLEAN}, \CEnum{ICET\_BYTE}, \CEnum{ICET\_SHORT},
\CEnum{ICET\_INT}, \CEnum{ICET\_FLOAT}, or \CEnum{ICET\_DOUBLE}

\subsubsection{Transferring Images}

Although the \CType{IceTImage} and \CType{IceTSparseImage} types are
opaque, they can be transferred as simple byte buffers.  To do so, you need
only the size of the buffer.  The following sends an image stored in the
variable \textC{image} of type \CType{IceTImage}.

\index{icetFullImageSize}
\index{icetGetImagePixelCount}
\index{icetAddSentBytes}
\index{ICET\_COMM\_SEND}
\begin{code}
  size = icetFullImageSize(icetGetImagePixelCount(image));
  icetAddSentBytes(size);
  ICET_COMM_SEND(image, size, ICET_BYTE, dest, tag);
\end{code}

And the following is the paired receive for the image.  Note that the
number of pixels in \textC{pixel\_count} need to be as large or larger then
the actual number of pixels sent, but it otherwise does not have to match
exactly.  And, of course, \textC{image} must be allocated (generally with
\CFunc{icetResizeBuffer} and \CFunc{icetReserveBufferMem}) with the
appropriate amount of memory.

\index{icetFullImageSize}
\index{ICET\_COMM\_RECV}
\begin{code}
  size = icetFullImageSize(pixels);
  ICET_COMM_RECV(image, size, ICET_BYTE, src, tag);
\end{code}

Most of the time, you will actually send compressed image data.  Compressed
images are sent in the same manner as full image.  The only difference is
to make sure you give the communication function the actual size of the
image.

\index{icetCompressImage}
\index{ICET\_COMM\_SEND}
\begin{code}
  size = icetCompressImage(image, compressed_image);
  ICET_COMM_SEND(compressed_image, size, ICET_BYTE, dest, tag);
\end{code}

And the following is the paired receive for the sparse image.  Note that we
do not need to know the actual number of bytes received.  Rather, we just
need to know the maximum size of the image and have a buffer that large.

\index{icetSparseImageSize}
\index{ICET\_COMM\_RECV}
\begin{code}
  size = icetSparseImageSize(pixels);
  ICET_COMM_RECV(compressed_image, size, ICET_BYTE, src tag);
\end{code}

\subsubsection{Helper Communication Functions}

\index{common.h}\textC{common.h} (found in the strategies directory)
contains some helper functions that implement common communication
patterns.  They may be helpful in implementing your strategy.

\label{manpage:icetRenderTransferFullImages}
\begin{Table}{3}
  \multicolumn{3}{l}{\textC{void }\CFunc{icetRenderTransferFullImages}\textC{(}}\\
  \makebox[2in]{}&\CType{IceTImage}&\CArg{imageBuffer}\textC{,}\\
  &\CType{IceTSparseImage}&\CArg{inImage}\textC{,}\\
  &\CType{IceTSparseImage}&\CArg{outImage}\textC{,}\\
  &\textC{GLint}&\CArg{num\_receiving}\textC{,}\\
  &\textC{GLint *}&\CArg{tile\_image\_dest}\quad\textC{);}
\end{Table}

\CFunc{icetRenderTransferFullImages} renders all the tiles that are
specified in the \CEnum{ICET\_CONTAINED\_TILES} state array and sends them
to the processors with ranks specified in \CArg{tile\_image\_dest}.  This
method is guaranteed not to deadlock.  It only uses memory given with the
buffer arguments, and will make its best efforts to get the graphics and
network hardware to run in parallel.

\CArg{imageBuffer} is a buffer big enough to hold color and/or depth values
that is \CEnum{ICET\_MAX\_PIXELS} big.  The size can be determined with the
\CFunc{icetFullImageSize} function in image.h.  \CArg{inImage} and
\CArg{outImage} are two buffers big enough to hold sparse color and depth
information for an image that is \CEnum{ICET\_MAX\_PIXELS} big.  The size
can be determined with the \CFunc{icetSparseImageSize} macro in
\index{image.h}\textC{image.h}.  \CArg{num\_receiving} is the number of
images this processor is receiving, and \CArg{tile\_image\_dest} is an
array where if tile $t$ is in \CEnum{ICET\_CONTAINED\_TILES}, then the
rendered image for tile $t$ is sent to $\CArg{tile\_image\_dest}[t]$.

There is also a more general form for transferring images or other large
data blocks.

\label{manpage:icetSendRecvLargeMessages}
\textC{typedef void *(*}\CType{IceTGenerateData}\textC{)(GLint id, GLint dest, GLint *size);}

\textC{typedef void *(*}\CType{IceTHandleData}\textC{)(void *, GLint src);}

\begin{Table}{3}
  \multicolumn{3}{l}{\textC{void }\CFunc{icetSendRecvLargeMessages}\textC{(}}\\
  \makebox[2in]{}&\textC{GLint}&\CArg{numMessagesSend}\textC{,}\\
  &\textC{GLint *}&\CArg{messageDestinations}\textC{,}\\
  &\textC{GLint}&\CArg{messagesInOrder}\textC{,}\\
  &\CType{IceTGenerateData}&\CArg{generateDataFunc}\textC{,}\\
  &\CType{IceTHandleData}&\CArg{handleDataFunc}\textC{,}\\
  &\textC{void *}&\CArg{incomingBuffer}\textC{,}\\
  &\textC{GLint}&\CArg{bufferSize}\textC{);}
\end{Table}

\CFunc{icetSendRecvLargeMessages} is similar to
\CFunc{icetRenderTransferFullImages} except that it works with generic
data, data generators, and data handlers.  It takes a count of a number of
messages to be sent and an array of ranks to send to.  Two callbacks are
required.  One generates the data (so large data may be generated JIT to
save memory) and the other handles incoming data.  The generate callback is
run right before the data it returns is sent to a particular destination.
This callback will not be called again until the memory it returned is no
longer in use, so the memory may be reused.  As large messages come in, the
handle callback is called.  As an optimization, if a process sends to
itself, then that will be the first message created.  This gives the
callback an opportunity to build its local data while waiting for incoming
data.  The handle callback returns a pointer to a buffer to be used for the
next large message receive.  It should be common for this message buffer to
be reused too.

\CArg{numMessagesSending} is a count of the number of large messages this
processor is sending out.  \CArg{messageDestinations} is an array of size
\CArg{numMessagesSending} that contains the ranks of message destinations.
\CArg{generateDataFunc} is a callback function that generates messages.
The function is given the index in \CArg{messageDestinations} and the rank
of the destination as arguments.  The data of the message and the size of
the message (in bytes) are returned.  The \CArg{generateDataFunc} will not
be called again until the returned data is no longer in use.  Thus the data
may be reused.  \CArg{handleDataFunc} is a callback function that processes
messages.  The function is given the data buffer and the rank of the
process that sent it.  The function is expected to return a buffer to use
for the next message receive.  If the callback is finished with the buffer
it was given, it is perfectly acceptable to return it again for reuse.
\CArg{incomingBuffer} is a buffer to use for the first incoming message.
\CArg{bufferSize} is the maximum size of a message.

\subsection{Internal Functions for Compositing}

In the strategies directory, the \index{common.h}\textC{common.h} header
has prototypes for the single image compositing algorithms described in the
Single Image Compositing section of this chapter.

\subsubsection{Parallel Compositing}

\label{manpage:icetTreeCompose}
\begin{Table}{3}
  \textC{void }\CFunc{icetTreeCompose}\textC{(}&\textC{GLint *}&\CArg{compose\_group}\textC{,}\\
  &\textC{GLint}&\CArg{group\_size}\textC{,}\\
  &\textC{GLint}&\CArg{image\_dest}\textC{,}\\
  &\CType{IceTImage}&\CArg{imageBuffer}\textC{,}\\
  &\CType{IceTSparseImage}&\CArg{compressedImageBuffer}\quad\textC{);}
\end{Table}

\CFunc{icetTreeCompose} performs a binary tree composition of images
amongst a subset of processes in the current communicator of the context.
Rather than perform the composition on all the processes in the
communicator, it performs them on a subset with arbitrary ordering. (Note
that ordering matters when doing alpha blending as opposed to the z-buffer
operation.)  \CArg{compose\_group} is the mapping of processes from the
communicator ranks to the ``group'' ranks.  The size of the groups (and the
length of the \CArg{compose\_group} array) is specified by
\CArg{group\_size}.  The compose image ends up in the processor with rank
$\CArg{compose\_group}[\CArg{image\_dest}]$.  \CArg{imageBuffer} should
contain the partial input image to be composited. (Of course, each process
should have its own partial image.  All processes should provide images of
identical dimensions.)  On the process with the rank
$\CArg{compose\_group}[\CArg{image\_dest}]$, the final image will be stored
in this buffer.  \CArg{compressedImageBuffer} is a buffer that is used
internally by \CFunc{icetTreeCompose} for compressing, sending, and
receiving images.  It must be large enough to hold an image as large as the
input, but its contents are ignored on the function invocation and the
contents are garbage on return.

The following is a very simple example of compositing the image on tile 0
and providing the result on the process with rank 0.  If ordered
compositing is enabled, then the order is respected.  This is similar to
the \index{strategy!serial}serial strategy except that only the first tile
is composited.

\begin{code}
IceTImage treeComposeTile0(void)
{
  GLint max_pixels;
  GLint rank;
  GLint num_proc;
  GLint *display_node;
  GLint image_dest;
  GLboolean ordered_composite;
  IceTImage image;
  IceTSparseImage scratchImage;
  GLint *compose_group;
  int i;

  icetGetIntegerv(ICET_NUM_TILES, &num_tiles);
  icetGetIntegerv(ICET_TILE_MAX_PIXELS, &max_pixels);
  icetGetIntegerv(ICET_RANK, &rank);
  icetGetIntegerv(ICET_NUM_PROCESSES, &num_proc);
  display_nodes = icetUnsafeStateGet(ICET_DISPLAY_NODES);
  ordered_composite = icetIsEnabled(ICET_ORDERED_COMPOSITE);

  icetResizeBuffer(  icetFullImageSize(max_pixels)
                   + icetSparseImageSize(max_pixels)
                   + sizeof(int)*num_proc);
  image         = icetReserveBufferMem(icetFullImageSize(max_pixels));
  scratchImage  = icetReserveBufferMem(icetSparseImageSize(max_pixels));
  compose_group = icetReserveBufferMem(sizeof(GLint)*num_proc);

  if (ordered_composite) {
    icetGetIntegerv(ICET_COMPOSITE_ORDER, compose_group);
    for (image_dest = 0; compose_group[image_dest] != display_nodes[i];
         image_dest++);
  } else {
    for (i = 0; i < num_proc; i++) {
      compose_group[i] = i;
    }
    image_dest = display_nodes[0];
  }

  icetGetTileImage(0, image);
  icetTreeCompose(compose_group, num_proc, image_dest, image, scratchImage);

  return image;
}
\end{code}

A much more scalable image compositing algorithm is binary swap.  Usually
you will use the binary-swap algorithm instead of tree compose.  The only
exception is that binary-swap has a bit more overhead than tree compose, so
for small amounts of processes it may be moderately faster to run tree
compose.

\label{manpage:icetBswapCompose}
\begin{Table}{3}
  \textC{void }\CFunc{icetBswapCompose}\textC{(}&\textC{GLint *}&\CArg{compose\_group}\textC{,}\\
  &\textC{GLint}&\CArg{group\_size}\textC{,}\\
  &\textC{GLint}&\CArg{image\_dest}\textC{,}\\
  &\CType{IceTImage}&\CArg{imageBuffer}\textC{,}\\
  &\CType{IceTSparseImage}&\CArg{scratchImage1}\textC{,}\\
  &\CType{IceTSparseImage}&\CArg{scratchImage2}\quad\textC{);}
\end{Table}

\CFunc{icetBswapCompose} behaves very much like \CFunc{icetTreeCompose}
except that it uses a different (and much more scalable) algorithm.  The
arguments of the two functions are very similar. (The only difference is
that \CFunc{icetBswapCompose} requires two \CType{IceTSparseImage} buffers
whereas \CFunc{icetTreeCompose} requires only one.) 

Rather than perform the composition on all the processes in the
communicator, \CFunc{icetBswapCompose} performs them on a subset with
arbitrary ordering. (Note that ordering matters when doing alpha blending
as opposed to the z-buffer operation.)  \CArg{compose\_group} is the
mapping of processes from the communicator ranks to the ``group'' ranks.
The size of the groups (and the length of the \CArg{compose\_group} array)
is specified by \CArg{group\_size}.  The compose image ends up in the
processor with rank $\CArg{compose\_group}[\CArg{image\_dest}]$.
\CArg{imageBuffer} should contain the partial input image to be
composited. (Of course, each process should have its own partial image.
All processes should provide images of identical dimensions.)  On the
process with the rank $\CArg{compose\_group}[\CArg{image\_dest}]$, the
final image will be stored in this buffer.  \CArg{scratchImage1} and
\CArg{scratchImage2} are buffers that are used internally by
\CFunc{icetBswapCompose} for compressing, sending, and receiving images.  It
must be large enough to hold an image as large as the input, but its
contents are ignored on the function invocation and the contents are
garbage on return.

The following is a very simple example of compositing the image on tile 0
and providing the result on the process with rank 0.  It is identical to
the previous example code except that it uses binary swap and is equally
similar to the \index{strategy!serial}serial strategy.  If ordered
compositing is enabled, then the order is respected.

\begin{code}
IceTImage bswapComposeTile0(void)
{
  GLint max_pixels;
  GLint rank;
  GLint num_proc;
  GLint *display_node;
  GLint image_dest;
  GLboolean ordered_composite;
  IceTImage image;
  IceTSparseImage scratchImage1, scratchImage2;
  GLint *compose_group;
  int i;

  icetGetIntegerv(ICET_NUM_TILES, &num_tiles);
  icetGetIntegerv(ICET_TILE_MAX_PIXELS, &max_pixels);
  icetGetIntegerv(ICET_RANK, &rank);
  icetGetIntegerv(ICET_NUM_PROCESSES, &num_proc);
  display_nodes = icetUnsafeStateGet(ICET_DISPLAY_NODES);
  ordered_composite = icetIsEnabled(ICET_ORDERED_COMPOSITE);

  icetResizeBuffer(  icetFullImageSize(max_pixels)
                   + icetSparseImageSize(max_pixels)*2
                   + sizeof(int)*num_proc);
  image         = icetReserveBufferMem(icetFullImageSize(max_pixels));
  scratchImage1 = icetReserveBufferMem(icetSparseImageSize(max_pixels));
  scratchImage2 = icetReserveBufferMem(icetSparseImageSize(max_pixels));
  compose_group = icetReserveBufferMem(sizeof(GLint)*num_proc);

  if (ordered_composite) {
    icetGetIntegerv(ICET_COMPOSITE_ORDER, compose_group);
    for (image_dest = 0; compose_group[image_dest] != display_nodes[i];
         image_dest++);
  } else {
    for (i = 0; i < num_proc; i++) {
      compose_group[i] = i;
    }
    image_dest = display_nodes[0];
  }

  icetGetTileImage(0, image);
  icetBswapCompose(compose_group, num_proc, image_dest, image,
                   scratchImage1, scrachImage2);

  return image;
}
\end{code}

\subsubsection{Local Compositing}

When developing a multi-tile compositing algorithm (or any parallel
compositing algorithm for that matter), it is often cannot be broken into
full composites of single images.  Instead, you must break the problem down
further into image transfers and image combinations.  Image transfers have
already been covered previously in this section.  The \IceT library
contains multiple methods to locally composite two images together.

\label{manpage:icetComposite}
\begin{Table}{3}
  \textC{void }\CFunc{icetComposite}\textC{(}&\CType{IceTImage}&\CArg{destBuffer}\textC{,}\\
  &\textC{const }\CType{IceTImage}&\CArg{srcBuffer}\textC{,}\\
  &\textC{int}&\CArg{srcOnTop}\quad\textC{);}
\end{Table}

\CFunc{icetComposite} takes the images stored in \CArg{destBuffer} and
\CArg{srcBuffer}, composites them together, and stores the result in
\CArg{destBuffer}.  The compositing operation is automatically determined
by the current state. (See
Chapter~\ref{sec:Customizing_Compositing:Compositing_Operation} for
information on how the compositing operation is determined.)  If the
compositing operation is order dependent, then the Boolean argument
\CArg{srcOnTop} determines whether \CArg{srcBuffer} or \CArg{destBuffer} is
on top.

If one of your images is compressed (stored in a \CType{IceTSparseImage},
it is faster to perform the compositing operation on the compressed image
rather than decompressing first.  In fact, it is faster to composite a
compressed image than two full image because the
\index{active-pixel~encoding}active-pixel encoding allows the composite
algorithm to skip over groups of background pixels.  This gives you the
double win of faster image transfer and faster compositing.

\label{manpage:icetCompressedComposite}
\begin{Table}{3}
  \multicolumn{3}{l}{\textC{void }\CFunc{icetCompressedComposite}\textC{(}}\\
  \makebox[2.5in]{}&\CType{IceTImage}&\CArg{destBuffer}\textC{,}\\
  &\textC{const }\CType{IceTSparseImage}&\CArg{srcBuffer}\textC{,}\\
  &\textC{int}&\CArg{srcOnTop}\quad\textC{);}
\end{Table}

\CFunc{icetCompressedComposite} behaves just like \CFunc{icetComposite}
except that \CArg{srcBuffer} is a compressed image rather than a full
image.  The images in \CArg{destBuffer} and \CArg{srcBuffer} are composited
together, and the results are stored in \CArg{destBuffer}.

Many parallel compositing algorithms break images into pieces, distribute
amongst processes, and composite the pieces.  To facilitate the compositing
image pieces, \IceT provides \CFunc{icetCompressedSubComposite}.

\label{manpage:icetCompressedSubComposite}
\begin{Table}{3}
  \multicolumn{3}{l}{\textC{void }\CFunc{icetCompressedSubComposite}\textC{(}}\\
  \makebox[2.5in]{}&\CType{IceTImage}&\CArg{destBuffer}\textC{,}\\
  &\textC{GLuint}&\CArg{offset}\textC{,}\\
  &\textC{GLuint}&\CArg{pixels}\textC{,}\\
  &\textC{const }\CType{IceTSparseImage}&\CArg{srcBuffer}\textC{,}\\
  &\textC{int}&\CArg{srcOnTop}\textC{);}
\end{Table}

The \CArg{destBuffer}, \CArg{srcBuffer} and \CArg{srcOnTop} arguments are
the same as those in \CFunc{icetCompressedComposite}.  The \CArg{offset}
and \CArg{pixels} arguments specify a region of contiguous pixels in
\CArg{destBuffer} to perform the compositing in.

\index{strategy|)}
