% -*- latex -*-

\chapter{Custom Array Storage}
\label{chap:Storage}

\index{array handle|(}

Chapters \ref{chap:BasicArrayHandles}, \ref{chap:FancyArrayHandles}, and \ref{chap:AccessingAllocatingArrays} introduce the \vtkmcont{ArrayHandle} class.
In them, we learned how an \textidentifier{ArrayHandle} manages the memory allocation of an array, provides access to the data via array portals, and supervises the movement of data between the control and execution environments.

\index{array handle!storage|(}
\index{storage|(}

In addition to these data management features, \textidentifier{ArrayHandle}
also provides a configurable \keyterm{storage} mechanism that allows you,
through efficient template configuration, to redefine how data are stored
and retrieved. The storage object provides an encapsulated interface around
the data so that any necessary strides, offsets, or other access patterns
may be handled internally. The relationship between array handles and their
storage object is shown in Figure~\ref{fig:ArrayHandleStorage}.

\begin{figure}[htb]
  \centering
  \includegraphics{images/ArrayHandleStorage}
  \caption{Array handles, storage objects, and the underlying data source.}
  \label{fig:ArrayHandleStorage}
\end{figure}

\index{array handle!storage!default}
\index{storage!default}

As previously discussed in Chapter \ref{chap:BasicArrayHandles},
\vtkmcont{ArrayHandle} takes two template arguments.
\begin{vtkmexample}{Declaration of the \protect\vtkmcont{ArrayHandle} templated class (again).}
template<typename T, typename StorageTag = VTKM_DEFAULT_STORAGE_TAG>
class ArrayHandle;
\end{vtkmexample}
The first argument is the only one required and has been demonstrated multiple times before.
The second (optional) argument specifies something called a storage, which provides the interface between the generic \vtkmcont{ArrayHandle} class and a specific storage mechanism in the control environment.
If the storage parameter is not explicitly defined, it is set to \vtkmmacro{VTKM\_DEFAULT\_STORAGE\_TAG}, which is a macro that resolves to \vtkmcont{StorageTagBasic}.

%% In this and the following sections we describe this storage mechanism. A
%% default storage is specified in much the same way as a default device
%% adapter is defined (as described in Section~\ref{sec:DefaultDeviceAdapter}).
%% It is done by setting the \vtkmmacro{VTKM\_STORAGE} macro. This macro must
%% be set before including any \VTKm header files. Currently the only
%% practical storage provided by \VTKm is the basic storage, which simply
%% allocates a continuous section of memory of the given base type. This
%% storage can be explicitly specified by setting \vtkmmacro{VTKM\_STORAGE} to
%% \vtkmmacro{VTKM\_STORAGE\_BASIC} although the basic storage will also be
%% used as the default if no other storage is specified (which is typical).

The default storage can always be overridden by specifying an array storage tag.
Here is an example of specifying the storage type when declaring an array handle.

\vtkmlisting[ex:ArrayHandleStorageParameter]{Specifying the storage type for an \textidentifier{ArrayHandle.}}{ArrayHandleStorageParameter.cxx}

Although setting an \textidentifier{ArrayHandle}'s storage explicitly to \textidentifier{StorageTagBasic} as in Example \ref{ex:ArrayHandleStorageParameter} is seldom useful (since this is the default value), setting the storage is a good way to propagate the storage mechanism through template parameters.
The remainder of this chapter uses the storage mechanism to customize the representation of arrays.

By replacing the storage template parameter for \textidentifier{ArrayHandle}, we can change how data are stored in memory.
For example, when storing \vtkm{Vec} objects, the basic storage writes all \vtkm{Vec}s sequentially in a single array.
\index{array of structures}
\index{ArrayHandle!array of structures}
This is known as an \keyterm{array of structures}.
An alternate representation would be to store each component of the \vtkm{Vec}s in a separate array.
\index{structure of arrays}
\index{ArrayHandle!structure of arrays}
This alternate layout is known as a \keyterm{structure of arrays}.
There are reasons one might want to represent vector data in a structure of arrays, and \VTKm provides a separate storage to implement that: \vtkmcont{StorageTagSOA}.
From an interface perspective, the two \textidentifier{ArrayHandle}s behave the same, but they have very different implementations.

As is typical of different types of storage, the basic and SOA storage types have convenience \textidentifier{ArrayHandle} subclasses: \vtkmcont{ArrayHandleBasic} and \vtkmcont{ArrayHandleSOA}, respectively.
These are trivial subclasses of \vtkmcont{ArrayHandle} with the appropriate
storage, but provide convenience methods for construction and data access.

One interesting consequence of using a generic storage object to manage
data within an array handle is that the storage can be defined
functionally rather than point to data stored in physical memory. Thus,
implicit array handles are easily created by adapting to functional
``storage.'' For example, the point coordinates of a uniform rectilinear
grid are implicit based on the topological position of the point. Thus, the
point coordinates for uniform rectilinear grids can be implemented as an
implicit array with the same interface as explicit arrays (where
unstructured grid points would be stored).
Many examples of this are listed in Chapter \ref{chap:FancyArrayHandles}.

In this chapter we explore the many ways you can manipulate the \textidentifier{ArrayHandle} storage.
There are many ways to create custom storage for \textidentifier{ArrayHandle}.
As we explore these different ways, we will start with the easiest but most restrictive ways and move to the most expressive ways.


\section{Implicit Array Handles}

\index{array handle!implicit|(}
\index{storage!implicit|(}
\index{implicit storage|(}
\index{implicit array handle|(}
\index{functional array|(}

The generic array handle and storage templating in \VTKm allows for
any type of operations to retrieve a particular value. Typically this is
used to convert an index to some location or locations in memory. However,
it is also possible to compute a value directly from an index rather than
look up some value in memory. Such an array is completely functional and
requires no storage in memory at all. Such a functional array is called an
\keyterm{implicit array handle}. Implicit arrays are an example of
\keyterm{fancy array handles}, which are array handles that behave like
regular arrays but do special processing under the covers to provide
values.

Specifying a functional or implicit array in \VTKm is straightforward.
\VTKm has a special class named \vtkmcont{ArrayHandleImplicit} that makes
an implicit array containing values generated by a user-specified
\keyterm{functor}. \index{functor} A functor is simply a C++ class or
struct that contains an overloaded parenthesis operator so that it can be
used syntactically like a function.

To demonstrate the use of \textidentifier{ArrayHandleImplicit}, let us say
we want an array of even numbers. The array has the values
$[0,2,4,6,\ldots]$ (double the index) up to some given size. Although we
could easily create this array in memory, we can save space and possibly
time by computing these values on demand.

\begin{didyouknow}
  \VTKm already comes with an implicit array handle named \vtkmcont{ArrayHandleCounting} that can make implicit even numbers as well as other more general counts.
  (See Section \ref{sec:CountingArrays} for details.)
  So in practice you would not have to create a special implicit array, but we are doing so here for demonstrative purposes.
\end{didyouknow}

The first step to using \textidentifier{ArrayHandleImplicit} is to declare
a functor. The functor's parenthesis operator should accept a single
argument of type \vtkm{Id} and return a value appropriate for that index.
The parenthesis operator should also be declared \textcode{const} because
it is not allowed to change the class' state.

\vtkmlisting{Functor that doubles an index.}{ImplicitArrayFunctor.cxx}

\begin{commonerrors}
  The functor used with \textidentifier{ArrayHandleImplicit} may contain state, but it must be trivially copiable across memory spaces.
  That means it cannot contain any virtual methods nor hold any pointers or references.
  It also means it cannot hold objects like an \textidentifier{ArrayHandle} or \textidentifier{ArrayPortal}.
  Such behavior may seem to work at first, but will quickly break down on different devices.
  The point of an implicit array is that it completely computes each value without referencing any external data.
  Later in this chapter we will explore many ways to create storage that access data stored in memory or other \textidentifier{ArrayHandle}s.
\end{commonerrors}

Once the functor is defined, an implicit array can be declared using the templated \vtkmcont{ArrayHandleImplicit} class.
The single template argument is the functor's type.

\vtkmlisting{Declaring a \textidentifier{ArrayHandleImplicit}.}{DeclareImplicitArray.cxx}

For convenience, \vtkmheader{vtkm/cont}{ArrayHandleImplicit.h} also
declares the \vtkmcont{make\_ArrayHandleImplicit} function. This function
takes a functor and the size of the array and returns the implicit array.

\vtkmlisting{Using \textidentifier{make\_ArrayHandleImplicit}.}{MakeArrayHandleImplicit.cxx}

\index{array handle!subclassing}

If the implicit array you are creating tends to be generally useful and is
something you use multiple times, it might be worthwhile to make a
convenience subclass of \vtkmcont{ArrayHandleImplicit} for your array.

\vtkmlisting[ex:ImplicitArrayHandleSubclass]{Custom implicit array handle for even numbers.}{ImplicitArrayHandle.cxx}

Subclasses of \textidentifier{ArrayHandle} provide constructors that
establish the state of the array handle. All array handle subclasses must
also use either the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro or the
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro. Both of these macros
define the types \classmember*{ArrayHandle}{Superclass}, \classmember*{ArrayHandle}{ValueType}, and
\classmember*{ArrayHandle}{StorageTag} as well as a set of constructors and operators
expected of all \textidentifier{ArrayHandle} classes. The difference
between these two macros is that \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS}
is used in templated classes whereas
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is used in non-templated
classes.

\label{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT}

The \textidentifier{ArrayHandle} subclass in Example~\ref{ex:ImplicitArrayHandleSubclass} is not templated, so it uses the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro.
This macro takes two parameters.
The first parameter is the name of the subclass where the macro is defined and the second parameter is the immediate superclass including the full template specification.
The second parameter of the macro must be enclosed in parentheses so that the C pre-processor correctly handles commas in the template specification.
(The other macro is described in Section~\ref{sec:VTKM_ARRAY_HANDLE_SUBCLASS} on page~\pageref{sec:VTKM_ARRAY_HANDLE_SUBCLASS}).

\index{functional array|)}
\index{implicit array handle|)}
\index{implicit storage|)}
\index{storage!implicit|)}
\index{array handle!implicit|)}

\section{Transformed Arrays}
\label{sec:TransformedArrays}

\index{array handle!transform|(}
\index{transformed array|(}

Another type of fancy array handle is the transformed array. A transformed
array takes another array and applies a function to all of the elements to
produce a new array. A transformed array behaves much like a map operation
except that a map operation writes its values to a new memory location
whereas the transformed array handle produces its values on demand so that
no additional storage is required.

Specifying a transformed array in \VTKm is straightforward. \VTKm has a
special class named \vtkmcont{ArrayHandleTransform} that takes an array
handle and a functor and provides an interface to a new array comprising
values of the functor applied the first array.

To demonstrate the use of \textidentifier{ArrayHandleTransform}, let us say
that we want to scale and bias all of the values in a target array. That
is, each value in the target array is going to be multiplied by a given
scale and then offset by adding a bias value. (The scale and bias are
uniform across all entries.) We could, of course, easily create a worklet
to apply this scale and bias to each entry in the target array and save the
result in a new array, but we can save space and possibly time by computing
these values on demand.

The first step to using \textidentifier{ArrayHandleTransform} is to declare
a functor. The functor's parenthesis operator should accept a single
argument of the type of the target array and return the transformed value.
For more generally applicable transform functors, it is often useful to
make the parenthesis operator a template. The parenthesis operator should
also be declared \textcode{const} because it is not allowed to change the
class' state.

\vtkmlisting{Functor to scale and bias a value.}{TransformArrayFunctor.cxx}

\begin{commonerrors}
  As with functors for implicit arrays, a functor for \textidentifier{ArrayHandleTransform} must be trivially copiable.
  It may not hold in its state reference to any other arrays.
  The functor may only access the one value from the input array that is passed as an argument to the parenthesis operator.
  Mechanisms to build arrays with more expressive access to one or more other arrays are described later in this chapter.
\end{commonerrors}

Once the functor is defined, a transformed array can be declared using the templated \vtkmcont{ArrayHandleTransform} class.
The first template argument is the type of array being transformed.
The second template argument is the type of functor used for the transformation.
The third template argument, which is optional, is the type for an inverse functor that provides the inverse operation of the functor in the second argument.
This inverse functor is used for writing values into the array.
For arrays that will only be read from, there is no need to supply this inverse functor.

That said, it is generally easier to use the \vtkmcont{make\_ArrayHandleTransform} convenience function.
This function takes an array and a functor (and optionally an inverse functor) and returns a transformed array.

\vtkmlisting{Using \textidentifier{make\_ArrayHandleTransform}.}{MakeArrayHandleTransform.cxx}

If the transformed array you are creating tends to be generally useful and
is something you use multiple times, it might be worthwhile to make a
convenience subclass of \vtkmcont{ArrayHandleTransform} or convenience
\textcode{make\_ArrayHandle*} function for your array.

\vtkmlisting[ex:TransformArrayHandleSubclass]{Custom transform array handle for scale and bias.}{TransformArrayHandle.cxx}

Subclasses of \textidentifier{ArrayHandle} provide constructors that
establish the state of the array handle. All array handle subclasses must
also use either the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro or the
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro. Both of these macros
define the types \classmember*{ArrayHandle}{Superclass}, \classmember*{ArrayHandle}{ValueType}, and
\classmember*{ArrayHandle}{StorageTag} as well as a set of constructors and operators
expected of all \textidentifier{ArrayHandle} classes. The difference
between these two macros is that \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS}
is used in templated classes whereas
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is used in non-templated
classes.

\label{sec:VTKM_ARRAY_HANDLE_SUBCLASS}

The \textidentifier{ArrayHandle} subclass in Example~\ref{ex:TransformArrayHandleSubclass} is templated, so it uses the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro.
This macro takes three parameters.
The first parameter is the name of the subclass where the macro is defined, the second parameter is the type of the subclass including the full template specification, and the third parameter is the immediate superclass including the full template specification.
The second and third parameters of the macro must be enclosed in parentheses so that the C pre-processor correctly handles commas in the template specification.
(The other macro is described in Section~\ref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT} on page~\pageref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT}).

\index{transformed array|)}
\index{array handle!transform|)}


\section{Decorated Arrays}
\label{sec:DecoratedArrays}

In the previous section, we saw how to augment a single array to modify its values in place.
However, \textidentifier{ArrayHandleTransform} is limited in that it only allows you to augment one array at a time, and it does not allow you to adjust the index lookups into the array.

If \textidentifier{ArrayHandleTransform} is not powerful enough, \VTKm provides \vtkmcont{ArrayHandleDecorator} for more general augmentation of arrays.
\textidentifier{ArrayHandleDecorator} provides a much more expressive (albeit complicated) interface.

In this section we will demonstrate the steps required to create a more general derived storage.
For the purposes of the example in this section, let us say we want 2 array handles to behave as one array with the contents interlaced together.
That is, the first $w$ items are the same as the first $w$ of the first array, the second $w$ items are the same as the first $w$ of the second array, the third $w$ items are the same as the second $w$ of the first array, the fourth $w$ items are the same as the second $w$ of the second array, and so on.
We could of course actually copy the data, but we can also do it in place.

Using \textidentifier{ArrayHandleDecorator} requires creating functors for getting and setting data in the arrays plus creating another ``interface'' structure to manage the functors and data.
As always, these can be used to define new \textidentifier{ArrayHandle} classes.

\subsection{Functors}

Like \textidentifier{ArrayHandleImplicit} and \textidentifier{ArrayHandleTransform}, you define the behavior of \textidentifier{ArrayHandleDecorator} by defining functors.
However, unlike the functors for the other \textidentifier{ArrayHandle}s, we relax the restrictions on \textidentifier{ArrayHandleDecorator}'s functors and allow them to hold array portals as part of its state.
Note that there are still restrictions on where these portals come from.
They are generated internally by \textidentifier{ArrayHandleDecorator} and are passed to the functors as described in Section \ref{sec:ArrayHandleDecoratorInterface}.

The decorator needs a functor with a parentheses operator that is given an index and returns a value for that index.
The interface is the same as that for \textidentifier{ArrayHandleImplicit}, but the returned result can come from data in array portals.

\vtkmlisting[ex:DecoratorFunctor.cxx]{Functor that interlaces two array portals.}{DecoratorFunctor.cxx}

This functor will be used for retrieving data.
If the array being generated is intended to be read-only, then this is all you need.
However, if you wish to create a writable array, then you need a second ``inverse'' functor to set data to the array.
Fundamentally, there is no reason why we should not support writing data to the interlaced arrays, so we define a secondary inverse functor.

\vtkmlisting[ex:DecoratorInverseFunctor.cxx]{Inverse functor for writing data to interlaced array portals.}{DecoratorInverseFunctor.cxx}

\subsection{Interface}
\label{sec:ArrayHandleDecoratorInterface}

The next step in creating a decorator array is to define an ``interface'' class that tells \textidentifier{ArrayHandleDecorator} how to read, write, and resize the array.
This is just a normal \textcode{class} or \textcode{struct} containing the following members.

\begin{description}
\item[\textcode{CreateFunctor}] %
  A method that takes one or more array portals and returns a functor used to get information from these portals.
  These portals will respectively come from the arrays being decorated.
  This method is required.
\item[\textcode{CreateInverseFunctor}] %
  A method that takes one or more array portals and returns a functor used to set information in these portals.
  These portals will respectively come from the arrays being decorated.
  This method is optional.
  If it is omitted, then the decorated array will be read-only.
\item[\textcode{AllocateSourceArrays}] %
  A method that resizes the arrays being decorated.
  The first three arguments are the new size of the array, a \vtkm{CopyFlag} indicating whether data should be preserved, and a \vtkmcont{Token}, respectively.
  The remaining arguments are the arrays being decorated.
  This method is optional.
  If it is omitted, then the decorated array cannot be resized after creation.
\end{description}

Our implementation of interlaced array handles produces the functors presented in Exercises \ref{ex:DecoratorFunctor.cxx} and \ref{ex:DecoratorInverseFunctor.cxx}.
It also provides a mechanism to resize the arrays.

\vtkmlisting{Decorator implementation class for interleaving \textidentifier{ArrayHandle}s.}{DecoratorImplementation.cxx}

\subsection{Subclass}

If the array decorator you are creating tends to be generally useful and is something you use multiple times, it might be worthwhile to make a convenience subclass of \vtkmcont{ArrayHandleDecorator} or convenience \textcode{make\_ArrayHandle*} function for your array.

\vtkmlisting[ex:DecoratorArrayHandle.cxx]{Custom decorator array handle for interleaving arrays.}{DecoratorArrayHandle.cxx}

Subclasses of \textidentifier{ArrayHandle} provide constructors that
establish the state of the array handle. All array handle subclasses must
also use either the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro or the
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro. Both of these macros
define the types \classmember*{ArrayHandle}{Superclass}, \classmember*{ArrayHandle}{ValueType}, and
\classmember*{ArrayHandle}{StorageTag} as well as a set of constructors and operators
expected of all \textidentifier{ArrayHandle} classes. The difference
between these two macros is that \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS}
is used in templated classes whereas
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is used in non-templated
classes.

The \textidentifier{ArrayHandle} subclass in Example \ref{ex:DecoratorArrayHandle.cxx} is templated, so it uses the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro.
This macro takes three parameters.
The first parameter is the name of the subclass where the macro is defined, the second parameter is the type of the subclass including the full template specification, and the third parameter is the immediate superclass including the full template specification.
The second and third parameters of the macro must be enclosed in parentheses so that the C pre-processor correctly handles commas in the template specification.
(The other macro is described in Section~\ref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT} on page~\pageref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT}).


\section{Derived Storage}
\label{sec:DerivedStorage}

\index{array handle!derived|(}
\index{storage!derived|(}
\index{derived storage|(}

A \keyterm{derived storage} is a type of fancy array that takes one or more
other arrays and changes their behavior in some way. A transformed array
(Section~\ref{sec:TransformedArrays}) is a specific type of derived array
with a simple mapping. In this section we will demonstrate the steps
required to create a more general derived storage. When applicable, it is
much easier to create a derived array as a transformed array or using the
other fancy arrays than to create your own derived storage. However, if
these pre-existing fancy arrays do not work work, for example if your
derivation uses multiple arrays or requires general lookups, you can do so
by creating your own derived storage. For the purposes of the example in
this section, let us say we want 2 array handles to behave as one array
with the contents interlaced together. That is, the first $w$ items are
the same as the first $w$ of the first array, the second $w$ items are
the same as the first $w$ of the second array, the third $w$ items are
the same as the second $w$ of the first array, the fourth $w$ items are
the same as the second $w$ of the second array, and so on. We could of
course actually copy the data, but we can also do it in place.

\subsection{Array Portal}

The first step to creating a derived storage is to build an array portal
that will take portals from arrays being derived. The portal must work in
both the control and execution environment.

Because the intention of our custom \textidentifier{ArrayHandle} is to augment the behavior of other \textidentifier{ArrayHandle}, it is typical for an \textidentifier{ArrayPortal} of this nature to reference other \textidentifier{ArrayPortal}s.

\vtkmlisting[ex:DerivedArrayPortal]{Derived array portal for concatenated arrays.}{DerivedArrayPortal.cxx}

\subsection{Storage}

The next step in creating a custom storage is to define a tag for the adapter.
We shall call ours \textcode{StorageTagInterlace} and it will be templated on the two array handle types that we are deriving.
Then, we need to create a specialization of the templated \vtkmcontinternal{Storage} class.
The \textidentifier{ArrayHandle} will instantiate an object using the array container tag we give it, and we define our own specialization so that it runs our interface into the code.

\vtkmcontinternal{Storage} has two template arguments: the
base type of the array and the storage tag.

\vtkmlisting{Prototype for \protect\vtkmcontinternal{Storage}.}{StoragePrototype.cxx}

The \vtkmcontinternal{Storage} must define the following items.
\newcommand*{\storageitems}{
  \begin{description}
  \item[{\classmember*{Storage}{ReadPortalType}}] %
    The type of an array portal that can be used to access the underlying data.
    This array portal needs only be read-only.
    That is, the \classmember*{ArrayPortal}{Set} method is optional.
  \item[{\classmember*{Storage}{WritePortalType}}] %
    The type of an array portal that can be used to access the underlying data.
    This array portal should be both read and write capable.
    If the storage is intended to be read-only, then \classmember*{Storage}{WritePortalType} should be left out and \vtkmmacro{VTKM\_STORAGE\_NO\_WRITE\_PORTAL} should be declared in the storage class instead.
  \item[{\classmember*{Storage}{CreateBuffers}}] %
    The actual data of an \textidentifier{ArrayHandle} is stored in a \textcode{std::vector} of \vtkmcontinternal{Buffer} objects.
    The \textidentifier{Storage} builds the array of \textidentifier{Buffer} objects in the \classmember*{Storage}{CreateBuffers} method.
    A derived array usually contains all the \textidentifier{Buffer}s from the \textidentifier{ArrayHandle}s it is referencing plus an additional \textidentifier{Buffer} for its own metadata.
    See Section \ref{sec:Buffer} for details on \textidentifier{Buffer}.
  \item[{\classmember*{Storage}{GetNumberOfValues}}] %
    This is a static method that takes a \textcode{std::vector} of \vtkmcontinternal{Buffer} objects and returns the number of values in the represented array.
    The number of values is either derived from the size of the \textidentifier{Buffer}s, the size of the arrays being referenced, or stored in the metadata.
  \item[{\classmember*{Storage}{ResizeBuffers}}] %
    This is a static method that takes a number of values for the arrays, a \textcode{std::vector} of \vtkmcontinternal{Buffer} objects, a \vtkm{CopyFlag}, and a \vtkmcont{Token}.
    It then resizes the memory in the \textidentifier{Buffer} objects to match the requested number of values.
    If the storage cannot be resized after it is created, then this method should be left out and the \vtkmmacro{VTKM\_STORAGE\_NO\_RESIZE} macro should be declared in the storage class instead.
  \item[{\classmember*{Storage}{CreateReadPortal}}] %
    This is a static method that takes a \textcode{std::vector} of \vtkmcontinternal{Buffer} objects, a \vtkmcont{DeviceAdapterId}, and a \vtkmcont{Token}.
    It returns a \classmember*{Storage}{ReadPortalType} that can be used to read the data on the given device.
  \item[{\classmember*{Storage}{CreateWritePortal}}] %
    This is a static method that takes a \textcode{std::vector} of \vtkmcontinternal{Buffer} objects, a \vtkmcont{DeviceAdapterId}, and a \vtkmcont{Token}.
    It returns a \classmember*{Storage}{WritePortalType} that can be used to read the data on the given device.
    If the storage is intended to be read-only, then \classmember*{Storage}{CreateWritePortal} should be left out and \vtkmmacro{VTKM\_STORAGE\_NO\_WRITE\_PORTAL} should be declared in the storage class instead.
  \end{description}
}
\storageitems

The \classmember*{Storage}{CreateBuffers} method must have a form that takes no arguments so that the \textidentifier{ArrayHandle} can initialize itself.
Although it is not necessary, it is often convenient to have overloads of \classmember*{Storage}{CreateBuffers} to create the \textcode{std::vector} of \vtkmcontinternal{Buffer} objects used for the storage based on some initial data (such as arrays it is based on).
The \vtkmcontinternal{CreateBuffers} function is often helpful for this task.
It will push all of its arguments into a \textcode{std::vector}.
Arguments that are \textidentifier{ArrayHandle}s or \textcode{std::vector}s of \textidentifier{Buffer}s will have their containing \textidentifier{Buffer}s add.
Arguments that do not contain any \textidentifier{Buffer} object will be added to the metadata of an empty \textidentifier{Buffer} object.

If the storage is working directly with data in memory, then the function of these items can be implemented through the \vtkmcontinternal{Buffer} objects.
However, if the storage is modifying the functionality of other array types, then the implementation is mostly performed using the storage for those respective array types.
The following example shows the implementation of \textidentifier{Storage} for our example derived array type.

\vtkmlisting[ex:DerivedArrayStorage]{\textidentifier{Storage} for derived container of interlaced arrays.}{DerivedArrayStorage.cxx}

\subsection{Subclass}

\index{array handle!subclassing}

The final step to make a derived storage is to create a mechanism to
construct an \textidentifier{ArrayHandle} with a storage derived from the
desired arrays. This can be done by creating a trivial subclass of
\vtkmcont{ArrayHandle} that simply constructs the array handle to the state
of an existing storage. It uses a protected constructor of
\vtkmcont{ArrayHandle} that accepts a constructed storage.

\vtkmlisting[ex:DerivedArrayHandle]{\textidentifier{ArrayHandle} for derived storage of concatenated arrays.}{DerivedArrayHandle.cxx}

Subclasses of \textidentifier{ArrayHandle} provide constructors that establish the state of the array handle.
All array handle subclasses must also use either the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro or the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro.
The difference between these two macros is that \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} is used in templated classes whereas \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is used in non-templated classes.
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} takes three arguments.
The first is the classname of the subclass being defined (without any template arguments).
The second argument is the fully resolved template with the template arguments given.
The third argument is the full name of the superclass, again with the template arguments specified.
The second and third arguments \emph{must} be encased in parentheses.
The \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is similar but only has 2 arguments: the derived class' name and the superclass.

Both of these macros define the following convenience types:
\begin{description}
\item[\classmember*{ArrayHandle}{Superclass}]
  The fully resolved type of the superclass (given as the last argument to \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} or \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT}).
\item[\classmember*{ArrayHandle}{ValueType}]
  The value type of the \textidentifier{ArrayHandle} (same as \textcode{Superclass::ValueType}).
\item[\classmember*{ArrayHandle}{StorageTag}]
  The storage tag of the \textidentifier{ArrayHandle} (same as \textcode{Superclass::StorageTag}).
\item[\classmember*{ArrayHandle}{StorageType}]
  The resolved template of \vtkmcontinternal{Storage}.
  This is useful for interacting with the \textidentifier{Storage} object appropriate for your \textidentifier{ArrayHandle}.
\item[\classmember*{ArrayHandle}{ReadPortalType}]
  The type of read portals for the \textidentifier{ArrayHandle} (same as \textcode{Superclass::ReadPortalType}).
\item[\classmember*{ArrayHandle}{WritePortalType}]
  The type of write portals for the \textidentifier{ArrayHandle} (same as \textcode{Superclass::WritePortalType}).
\end{description}
The macros also provide a set of constructors and operators expected of all \textidentifier{ArrayHandle} classes.

The \textidentifier{ArrayHandle} subclass in
Example~\ref{ex:DerivedArrayHandle} is templated, so it uses the
\vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro. (The other macro is
described in Section~\ref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT} on
page~\pageref{sec:VTKM_ARRAY_HANDLE_SUBCLASS_NT}). This macro takes three
parameters. The first parameter is the name of the subclass where the macro
is defined, the second parameter is the type of the subclass including the
full template specification, and the third parameter is the immediate
superclass including the full template specification. The second and third
parameters of the macro must be enclosed in parentheses so that the C
pre-processor correctly handles commas in the template specification.

It is also customary to create helper functions for creating \textidentifier{ArrayHandle}s.
This makes it simpler than creating matching template interfaces.

\vtkmlisting{Helper function for creating a custom derived \textidentifier{ArrayHandle}.}{DerivedMakeArrayHandle.cxx}

\vtkmcont{ArrayHandleCompositeVector} is an example of a derived array
handle provided by \VTKm. It references some fixed number of other arrays,
pulls a specified component out of each, and produces a new component that
is a tuple of these retrieved components.

\index{derived storage|)}
\index{storage!derived|)}
\index{array handle!derived|)}

\index{array handle!fancy|)}
\index{fancy array handle|)}



\section{Adapting Data Structures}
\label{sec:ArrayHandle:Adapting}

\index{array handle!adapting|(}
\index{storage!adapting|(}

The intention of the storage parameter for \vtkmcont{ArrayHandle} is to
implement the strategy design pattern to enable \VTKm to interface directly
with the data of any third party code source. \VTKm is designed to work
with data originating in other libraries or applications. By creating a new
type of storage, \VTKm can be entirely adapted to new kinds of data
structures.

\begin{didyouknow}
  \VTKm comes with several types of \textidentifier{ArrayHandle} that can adapt memory in different ways.
  In practice, it is rare to have to write a custom \textidentifier{ArrayHandle} to adapt to a data structure, and this example is particularly contrived.
  However, we document it here for completeness.
\end{didyouknow}

\begin{commonerrors}
  Keep in mind that memory layout used can have an effect on the running
  time of algorithms in \VTKm. Different data layouts and memory access can
  change cache performance and introduce memory affinity problems. The
  example code given in this section will likely have poorer cache
  performance than the basic storage provided by \VTKm. However, that might
  be an acceptable penalty to avoid data copies.
\end{commonerrors}

In this section we demonstrate the steps required to adapt the array handle
to a data structure provided by a third party. For the purposes of the
example, let us say that some fictitious library named ``foo'' has a simple
structure named \textcode{FooAttributes} that holds the field values for a
particular part of a mesh, and then maintain the field values for all
locations in a mesh in a \textcode{FooFields} object.

\vtkmlisting{Fictitious field storage used in custom array storage examples.}{FictitiousFieldStorage.cxx}

There are few restrictions on the structure of the data.
The only real restriction is that you must be able to get the data in buffers of raw C pointers containing trivially copiable objects.
In this example, we note that \textcode{FooFields} can return an array of \textcode{FooAttributes}, which satisfies this requirement.

\VTKm expects separate arrays for each of the fields rather than a single
array containing a structure holding all of the fields. However, rather
than copy each field to its own array, we can create a storage for each
field that points directly to the data in a \textcode{FooFields}
object.

\subsection{Buffer Objects}
\label{sec:Buffer}

\VTKm manages data across devices using the \vtkmcontinternal{Buffer} object.
As its name implies, \textidentifier{Buffer} manages a buffer in memory. This block of bytes can be allocated on different devices and the data it contains will be transferred across them.
The \textidentifier{Buffer} object operates by specifying its size and then requesting pointers to the reserved memory on different devices.

\textidentifier{Buffer} contains the following methods.

\begin{description}
\item[{\classmember*{Buffer}{GetNumberOfBytes}}] %
  Returns the number of bytes held by the buffer.
  \textidentifier{Buffer} actually allocates memory lazily, so there might not actually be any memory allocated anywhere.
  It is also possible that memory is simultaneously allocated on multiple devices.
  The number of bytes is returned as a \vtkm{BufferSizeType}, which might be a larger integer than \vtkm{Id}.
\item[{\classmember*{Buffer}{SetNumberOfBytes}}] %
  Changes the size of the buffer.
  \classmember*{Buffer}{SetNumberOfBytes} has three arguments.
  The first argument is the number of bytes to allocate.
  The second argument is a \vtkm{CopyFlag} that indicates whether any existing data in the buffer should be preserved.
  The third argument is a \vtkmcont{Token} that ensures that the resize will not interfere with other operations happening on the \textidentifier{Buffer}'s data.
\item[{\classmember*{Buffer}{IsAllocatedOnHost}}] %
  Returns true if the \textidentifier{Buffer} has memory allocated on the host (for the control environment).
\item[{\classmember*{Buffer}{IsAllocatedOnDevice}}] %
  Returns true if the \textidentifier{Buffer} has memory allocated on the device specified by a given device adapter tag.
  If \vtkmcont{DeviceAdapterTagAny} is given as the device, then this returns true if the \textidentifier{Buffer} is allocated on any device.
  If \vtkmcont{DeviceAdapterTagUnknown} is given as the device, then this returns true if the \textidentifier{Buffer} is allocated on the host (same as \classmember*{Buffer}{IsAllocatedOnHost}).
\item[{\classmember*{Buffer}{ReadPointerHost}}] %
  Returns a readable host (control environment) pointer to the buffer.
  Memory will be allocated and data will be copied as necessary.
  A \vtkmcont{Token} object is passed to \classmember*{Buffer}{ReadPointerHost}, and the memory at the pointer will be valid as long as the \textidentifier{Token} is still in scope.
  Any write operation to this buffer will be blocked until then.
\item[{\classmember*{Buffer}{ReadPointerDevice}}] %
  Returns a readable device pointer to the buffer.
  The first argument to \classmember*{Buffer}{ReadPointerDevice} is a \vtkmcont{DeviceAdapterId}, and the pointer returned will only be valid for this device.
  If the device is \vtkmcont{DeviceAdapterTagUnknown}, then this method has the same behavior as \classmember*{Buffer}{ReadPointerHost}.
  Memory will be allocated and data will be copied as necessary.
  A \vtkmcont{Token} object is passed to \classmember*{Buffer}{ReadPointerDevice}, and the memory at the pointer will be valid as long as the \textidentifier{Token} is still in scope.
  Any write operation to this buffer will be blocked until then.
\item[{\classmember*{Buffer}{WritePointerHost}}] %
  Returns a writable host (control environment) pointer to the buffer.
  Memory will be allocated and data will be copied as necessary.
  A \vtkmcont{Token} object is passed to \classmember*{Buffer}{WritePointerHost}, and the memory at the pointer will be valid as long as the \textidentifier{Token} is still in scope.
  Any read or write operation to this buffer will be blocked until then.
\item[{\classmember*{Buffer}{WritePointerDevice}}] %
  Returns a writable device pointer to the buffer.
  The first argument to \classmember*{Buffer}{WritePointerDevice} is a \vtkmcont{DeviceAdapterId}, and the pointer returned will only be valid for this device.
  If the device is \vtkmcont{DeviceAdapterTagUnknown}, then this method has the same behavior as \classmember*{Buffer}{WritePointerHost}.
  Memory will be allocated and data will be copied as necessary.
  A \vtkmcont{Token} object is passed to \classmember*{Buffer}{WritePointerDevice}, and the memory at the pointer will be valid as long as the \textidentifier{Token} is still in scope.
  Any read or write operation to this buffer will be blocked until then.
\item[{\classmember*{Buffer}{DeepCopyFrom}}] %
  Copies the data from the provided buffer into this buffer.
  If a device is given, then the copy will be preferred for that device.
\item[{\classmember*{Buffer}{ReleaseDeviceResources}}] %
  Unallocates the buffer from all devices.
  This method preserves the data on the host even if the data must be transferred there.
\item[{\classmember*{Buffer}{SetMetaData}}] %
  Takes an arbitrary object and copies it to the metadata of this buffer.
  Any existing metadata is deleted.
  Any object can be used as metadata as long as the object has a default constructor and is copiable.
  Holding metadata in a \textidentifier{Buffer} is optional, but it can be helpful for storing additional information or objects that cannot be implied by the buffer itself.
\item[{\classmember*{Buffer}{GetMetaData}}] %
  Gets the metadata for the buffer.
  When you call this method, you have to specify a template parameter for the type of the metadata.
  If the metadata has not yet been set in this buffer, a new metadata object is created, set to this buffer, and returned.
  If metadata of a different type has already been set, then an exception is thrown.
  The returned value is a reference that can be manipulated to alter the metadata of the buffer.
\item[{\classmember*{Buffer}{HasMetaData}}] %
  Returns whether the \textidentifier{Buffer} holds metadata.
\item[{\classmember*{Buffer}{MetaDataIsType}}] %
  Determines if the metadata for the buffer is set to a particular type.
  Specify the type of metadata as a template argument.
\end{description}

In addition to using \textidentifier{Buffer} to allocate data on devices and the host, you can wrap a \textidentifier{Buffer} around data that is already allocated.
This is done by using the \vtkmcontinternal{MakeBuffer} function.
This method takes 6 arguments: the \vtkmcont{DeviceAdapterId} of where the memory is allocated (\vtkmcont{DeviceAdapterTagUnknown} if on the host), the pointer to the memory buffer, a pointer to a container managing the buffer, the size of the buffer in bytes, a function used to delete the buffer, and a function used to deallocate the buffer.
An example of using \textidentifier{MakeBuffer} is given later.

\subsection{Array Portal}

The first step in creating an adapter storage is to create an array portal to the data.
This is described in more detail in Section~\ref{sec:ArrayPortals} and is generally straightforward for simple containers like this, which simple set and get values in a C array.
Here is an example implementation for our \textcode{FooFields} container.

\vtkmlisting[ex:ArrayPortalAdapter]{Array portal to adapt a third-party container to \VTKm.}{ArrayPortalAdapter.cxx}

\subsection{Storage}

The next step in creating an adapter storage is to define a tag for the
adapter. We shall call ours
\textcode{StorageTagFooPressure}. Then, we need to create a
specialization of the templated \vtkmcontinternal{Storage}
class. The \textidentifier{ArrayHandle} will instantiate an object using
the array container tag we give it, and we define our own specialization so
that it runs our interface into the code.

\vtkmcontinternal{Storage} has two template arguments: the
base type of the array and the storage tag.

\vtkmlisting{Prototype for \protect\vtkmcontinternal{Storage}.}{StoragePrototype.cxx}

The \vtkmcontinternal{Storage} must define the following items.
\storageitems

The following provides an example implementation of our adapter to \textcode{FooFields}.
It relies on the \textcode{ArrayPortalFooPressure} provided in Example~\ref{ex:ArrayPortalAdapter}.

\vtkmlisting{Storage to adapt a third-party container to \VTKm.}{StorageAdapter.cxx}

One interesting feature of this example is the \classmember*{Storage}{CreateBuffers} method on line \ref{ex:StorageAdapter.cxx:create-buffers} that wraps the \textidentifier{Buffer} around an existing \textcode{FooFields} object.
This allows \VTKm to read and write directly to and from the memory shared with the \textcode{FooFields}.
To make this work, \classmember*{Storage}{CreateBuffers} creates a new \vtkmcontinternal{Buffer} object using the \vtkmcontinternal{MakeBuffer} function (lines \ref{ex:StorageAdapter.cxx:make-buffer-start}--\ref{ex:StorageAdapter.cxx:make-buffer-end}).
The first 4 arguments are straightforward: device holding the memory, the actual memory, the object containing the memory, and the number of bytes in the buffer.
The fifth and sixth arguments are function pointers that handle deleting and reallocating the buffer (in the control environment).
These functions can be defined as follows.

\vtkmlisting{Memory handling functions to adapt a third-party data structure to \textidentifier{ArrayHandle}.}{ArrayHandleAdapterMemManage.cxx}

\subsection{Subclass}

\index{array handle!subclassing}

The final step to make a storage adapter is to make a mechanism to
construct an \textidentifier{ArrayHandle} that points to a particular
storage. This can be done by creating a trivial subclass of
\vtkmcont{ArrayHandle} that simply constructs the array handle to the state
of an existing container.

\vtkmlisting[ex:ArrayHandleAdapter]{Array handle to adapt a third-party container to \VTKm.}{ArrayHandleAdapter.cxx}

%% Subclasses of \textidentifier{ArrayHandle} provide constructors that
%% establish the state of the array handle. All array handle subclasses must
%% also use either the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS} macro or the
%% \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro. Both of these macros
%% define the types \classmember*{ArrayHandle}{Superclass}, \classmember*{ArrayHandle}{ValueType}, and
%% \classmember*{ArrayHandle}{StorageTag} as well as a set of constructors and operators
%% expected of all \textidentifier{ArrayHandle} classes. The difference
%% between these two macros is that \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS}
%% is used in templated classes whereas
%% \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} is used in non-templated
%% classes.

%% The \textidentifier{ArrayHandle} subclass in
%% Example~\ref{ex:ArrayHandleAdapter} is not templated, so it uses
%% the \vtkmmacro{VTKM\_ARRAY\_HANDLE\_SUBCLASS\_NT} macro. (The other macro
%% is described in Section~\ref{sec:VTKM_ARRAY_HANDLE_SUBCLASS} on
%% page~\pageref{sec:VTKM_ARRAY_HANDLE_SUBCLASS}). This macro takes two
%% parameters. The first parameter is the name of the subclass where the macro
%% is defined and the second parameter is the immediate superclass including
%% the full template specification. The second parameter of the macro must be
%% enclosed in parentheses so that the C pre-processor correctly handles
%% commas in the template specification.

With this new version of \textidentifier{ArrayHandle}, \VTKm can now read
to and write from the \textcode{FooFields} structure directly.

\vtkmlisting[ex:UsingArrayHandleAdapter]{Using an \textidentifier{ArrayHandle} with custom container.}{UsingArrayHandleAdapter.cxx}

\begin{commonerrors}
  When using an \textidentifier{ArrayHandle} in \VTKm some code may be executed in an execution environment with a different memory space.
  In these cases data written to an \textidentifier{ArrayHandle} with a custom storage will not be written directly to the storage system you defined.
  Rather, they will be written to a separate array in the execution environment.
  If you need to access data in your custom data structure, make sure you call \classmember*{ArrayHandle}{SyncControlArray} on the \textidentifier{ArrayHandle}, as is demonstrated in Example~\ref{ex:UsingArrayHandleAdapter}.
\end{commonerrors}

\index{array handle!storage!default}
\index{storage!default}

\begin{commonerrors}
  One of the challenges of introducing a new storage for \textidentifier{ArrayHandle} is that if the rest of the \VTKm does not recognize the new \textidentifier{ArrayHandle}, they will not be built with compile support for it.
  Thus, if you do make a custom data adapter for \textidentifier{ArrayHandle}, you will likely need to define a new set of default types that adds the new storage to \vtkmmacro{VTKM\_DEFAULT\_STORAGE\_LIST}.
  \fix{Reference chapter on defining default types?}
\end{commonerrors}

\index{storage!adapting|)}
\index{array handle!adapting|)}



\index{storage|)}
\index{array handle!storage|)}

\index{array handle|)}
