% -*- latex -*-

\chapter{Unknown Array Handles}
\label{chap:UnknownArrayHandle}

\index{unknown array handle|(}
\index{array handle!unknown|(}

The \textidentifier{ArrayHandle} class uses templating to make very
efficient and type-safe access to data. However, it is sometimes
inconvenient or impossible to specify the element type and storage at
run-time. The \textidentifier{UnknownArrayHandle} class provides a mechanism
to manage arrays of data with unspecified types.

\vtkmcont{UnknownArrayHandle} holds a reference to an array. Unlike
\textidentifier{ArrayHandle}, \textidentifier{UnknownArrayHandle} is
\emph{not} templated. Instead, it uses C++ run-type type information to
store the array without type and cast it when appropriate.

\index{unknown array handle!construct}
An \textidentifier{UnknownArrayHandle} can be established by constructing it
with or assigning it to an \textidentifier{ArrayHandle}. The following
example demonstrates how an \textidentifier{UnknownArrayHandle} might be
used to load an array whose type is not known until run-time.

\vtkmlisting{Creating an \textidentifier{UnknownArrayHandle}.}{CreateUnknownArrayHandle.cxx}

\section{Allocation}
\label{sec:UnknownArrayHandleAllocation}
\index{unknown array handle!allocation}
Data pointed to by an \textidentifier{UnknownArrayHandle} is not directly accessible.
However, it is still possible to do some type-agnostic manipulation of the array allocations.

First, it is always possible to call \classmember{UnknownArrayHandle}{GetNumberOfValues} to retrieve the current size of the array.
It is also possible to call \classmember{UnknownArrayHandle}{Allocate} to change the size of an unknown array.
\textidentifier{UnknownArrayHandle}'s \classmember*{UnknownArrayHandle}{Allocate} works exactly the same as the \classmember*{ArrayHandle}{Allocate} in the basic \textidentifier{ArrayHandle}.

\vtkmlisting{Checking the size of an \textidentifier{ArrayHandle} and resizing it.}{UnknownArrayHandleResize.cxx}

It is often the case where you have an \textidentifier{UnknownArrayHandle} as the input to an operation and you want to generate an output of the same type.
To handle this case, use the \classmember*{UnknownArrayHandle}{NewInstance} method to create a new array of the same type (without having to determine the type).

\vtkmlisting{Creating a new instance of an unknown array handle.}{NonTypeUnknownArrayHandleNewInstance.cxx}

That said, there are many fancy array types (described in Chapter \ref{chap:FancyArrayHandles}) that cannot be used as outputs.
Thus, if you do not know the storage of the array, the similar array returned by \classmember*{UnknownArrayHandle}{NewInstance} could be infeasible for use as an output.
Thus, \textidentifier{UnknownArrayHandle} also contains the \classmember*{UnknownArrayHandle}{NewInstanceBasic} method to create a new array with the same value type but using the basic array storage, which can always be resized and written to.

\vtkmlisting{Creating a new basic instance of an unknown array handle.}{UnknownArrayHandleBasicInstance.cxx}

It is occasionally the case that you need a new array of a similar type, but that type has to hold floating point values.
For example, if you had an operation that computed a discrete cosine transform on an array, the result would be very inaccurate if stored as integers.
In this case, you would actually want to store the result in an array of floating point values.
For this case, you can use the \classmember*{UnknownArrayHandle}{NewInstanceFloatBasic} to create a new basic \textidentifier{ArrayHandle} with the component type changed to \vtkm{FloatDefault}.
For example, if the \textidentifier{UnknownArrayHandle} stores an \textidentifier{ArrayHandle} of type \vtkm{Id}, \classmember*{UnknownArrayHandle}{NewInstanceFloatBasic} will create an \textidentifier{ArrayHandle} of type \vtkm{FloatDefault}.
If the \textidentifier{UnknownArrayHandle} stores an \textidentifier{ArrayHandle} of type \vtkm{Id3}, \classmember*{UnknownArrayHandle}{NewInstanceFloatBasic} will create an \textidentifier{ArrayHandle} of type \vtkm{Vec3f}.

\vtkmlisting{Creating a new array instance with floating point values.}{UnknownArrayHandleFloatInstance.cxx}

\section{Casting to Known Types}
\label{sec:UnknownArrayHandle:CastToKnownType}

\index{unknown array handle!cast|(}
Data pointed to by an \textidentifier{UnknownArrayHandle} is not directly
accessible.
To access the data, you need to retrieve the data as an \textidentifier{ArrayHandle}.
If you happen to know (or can guess) the type, you can use the \classmember*{UnknownArrayHandle}{AsArrayHandle} method to retrieve the array as a specific type.

\vtkmlisting{Retrieving an array of a known type from \textidentifier{UnknownArrayHandle}.}{UnknownArrayHandleAsArrayHandle1.cxx}

\classmember*{UnknownArrayHandle}{AsArrayHandle} actually has two forms.
The first form, shown in the previous example, has no arguments and returns the \textidentifier{ArrayHandle}.
This form requires you to specify the type of array as a template parameter.
The alternate form has you pass a reference to a concrete \textidentifier{ArrayHandle} as an argument as shown in the following example.
This form can imply the template parameter from the argument.

\vtkmlisting{Alternate form for retrieving an array of a known type from \textidentifier{UnknownArrayHandle}.}{UnknownArrayHandleAsArrayHandle2.cxx}

\classmember*{UnknownArrayHandle}{AsArrayHandle} treats \textidentifier{ArrayHandleCast} and \textidentifier{ArrayHandleMultiplexer} special.
If the special \textidentifier{ArrayHandle} can hold the actual array stored, then \classmember*{UnknownArrayHandle}{AsArrayHandle} will return successfully.
In the following example, \classmember*{UnknownArrayHandle}{AsArrayHandle} returns an array of type \vtkm{Float32} as an \textidentifier{ArrayHandleCast} that converts the values to \vtkm{Float64}.

\vtkmlisting{Getting a cast array handle from an \textidentifier{ArrayHandleCast}.}{UnknownArrayHandleAsCastArray.cxx}

\begin{didyouknow}
  The inverse retrieval works as well.
  If you create an \textidentifier{UnknownArrayHandle} with an \textidentifier{ArrayHandleCast} or \textidentifier{ArrayHandleMultiplexer}, you can get the underlying array with \classmember*{UnknownArrayHandle}{AsArrayHandle}.
  These relationships also work recursively (e.g. an array placed in a cast array which is placed in a multiplexer).
\end{didyouknow}

\index{unknown array handle!query type|(}

If the \textidentifier{UnknownArrayHandle} cannot store its array in the type given to \classmember*{UnknownArrayHandle}{AsArrayHandle}, it will throw an exception.
Thus, you should not use \classmember*{UnknownArrayHandle}{AsArrayHandle} with types that you are not sure about.
Use the \classmember*{UnknownArrayHandle}{CanConvert} method to determine if a given \textidentifier{ArrayHandle} type will work with \classmember*{UnknownArrayHandle}{AsArrayHandle}.

\vtkmlisting{Querying whether a given \textidentifier{ArrayHandle} can be retrieved from an \textidentifier{UnknownArrayHandle}.}{UnknownArrayHandleCanConvert.cxx}

\index{unknown array handle!cast|)}

By design, \classmember*{UnknownArrayHandle}{CanConvert} will return true for types that are not actually stored in the \textidentifier{UnknownArrayHandle} but can be retrieved.
If you need to know specifically what type is stored in the \textidentifier{UnknownArrayHandle}, you can use the \classmember*{UnknownArrayHandle}{IsType} method instead.
If you need to query either the value type or the storage, you can use \classmember*{UnknownArrayHandle}{IsValueType} and \classmember*{UnknownArrayHandle}{IsStorageType}, respectively.
\textidentifier{UnknownArrayHandle} also provides \classmember*{UnknownArrayHandle}{GetValueTypeName} and \classmember*{UnknownArrayHandle}{GetStorageTypeName} for debugging purposes.

\begin{commonerrors}
  \classmember*{UnknownArrayHandle}{CanConvert} is almost always safer to use than \classmember*{UnknownArrayHandle}{IsType} or its similar methods.
  Even though \classmember*{UnknownArrayHandle}{IsType} reflects the actual array type, \classmember*{UnknownArrayHandle}{CanConvert} better describes how \textidentifier{UnknownArrayHandle} will behave.
\end{commonerrors}

\index{unknown array handle!query type|)}

\index{unknown array handle!copy|(}

If you do not know the exact type of the array contained in an \textidentifier{UnknownArrayHandle}, a brute force method to get the data out is to copy it to an array of a known type.
This can be done with the \classmember{UnknownArrayHandle}{DeepCopyFrom} method, which will copy the contents of a target array into an existing array of a (potentially) different type.

\vtkmlisting{Deep copy arrays of unknown types.}{UnknownArrayHandleDeepCopy.cxx}

It is often the case that you have good reason to believe that an array is of an expected type, but you have no way to be sure.
To simplify code, the most rational thing to do is to get the array as the expected type if that is indeed what it is, or to copy it to an array of that type otherwise.
The \classmember{UnknownArrayHandle}{CopyShallowIfPossible} does just that.

\vtkmlisting{Using \textidentifier{ArrayCopyShallowIfPossible} to get an unknown array as a particular type.}{UnknownArrayHandleShallowCopy.cxx}

\begin{didyouknow}
  The \textidentifier{UnknownArrayHandle} copy methods behave similarly to the \vtkmcont{ArrayCopy} functions.
  One advantage of using the \textidentifier{UnknownArrayHandle} methods is that they do not require using a device compiler (such as \textfilename{nvcc}).
  Both versions will (potentially) perform the copy on a device, but the methods for \textidentifier{UnknownArrayHandle} are sufficiently hidden in a library to avoid calling code needing to compile device instructions.
\end{didyouknow}

\index{unknown array handle!copy|)}

\section{Casting to a List of Potential Types}
\label{sec:UnknownArrayHandle:CastToPotentialTypes}

\index{unknown array handle!cast|(}

Using \classmember*{UnknownArrayHandle}{AsArrayHandle} is fine as long as the correct types are known, but often times they are not.
For this use case \textidentifier{UnknownArrayHandle} has a method named \classmember*{UnknownArrayHandle}{CastAndCallForTypes} that attempts to cast the array to some set of types.

The \classmember*{UnknownArrayHandle}{CastAndCallForTypes} method accepts a functor to run on the appropriately cast array.
The functor must have an overloaded const parentheses operator that accepts an \textidentifier{ArrayHandle} of the appropriate type.
You also have to specify two template parameters that specify a \vtkm{List} of value types to try and a \vtkm{List} of storage types to try, respectively.
The macros \vtkmmacro{VTKM\_DEFAULT\_TYPE\_LIST} and \vtkmmacro{VTKM\_DEFAULT\_STORAGE\_LIST} are often used when nothing more specific is known.

\vtkmlisting[ex:UsingCastAndCallForTypes]{Operating on an \textidentifier{UnknownArrayHandle} with \textcode{CastAndCallForTypes}.}{UsingCastAndCallForTypes.cxx}

\begin{didyouknow}
  The first (required) argument to \classmember*{UnknownArrayHandle}{CastAndCallForTypes} is the functor to call with the array.
  You can supply any number of optional arguments after that.
  Those arguments will be passed directly to the functor.
  This makes it easy to pass state to the functor.
\end{didyouknow}

\begin{didyouknow}
  When an \textidentifier{UnknownArrayHandle} is used in place of an \textidentifier{ArrayHandle} as an argument to a worklet invocation, it will internally use \classmember*{UnknownArrayHandle}{CastAndCallForTypes} to attempt to call the worklet with an \textidentifier{ArrayHandle} of the correct type.
\end{didyouknow}

\textidentifier{UnknownArrayHandle} has a simple subclass named \vtkmcont{UncertainArrayHandle} for use when you can narrow the array to a finite set of types.
\textidentifier{UncertainArrayHandle} has two template parameters that must be specified: a \vtkm{List} of value types and a \vtkm{List} of storage types.
\textidentifier{UncertainArrayHandle} has a method named \classmember*{UncertainArrayHandle}{CastAndCall} that behaves the same as \classmember*{UnknownArrayHandle}{CastAndCallForTypes} except that you do not have to specify the types to try.
Instead, the types are taken from the template parameters of the \textidentifier{UncertainArrayHandle} itself.

\vtkmlisting{Using \textidentifier{UncertainArrayHandle} to cast and call a functor.}{UncertainArrayHandle.cxx}

\begin{didyouknow}
  Like with \textidentifier{UnknownArrayHandle}, if an \textidentifier{UncertainArrayHandle} is used in a worklet invocation, it will internally use \classmember*{UncertainArrayHandle}{CastAndCall}.
  This provides a convenient way to specify what array types the invoker should try.
\end{didyouknow}

Both \textidentifier{UnknownArrayHandle} and \textidentifier{UncertainArrayHandle} provide a method named \classmember*{UnknownArrayHandle}{ResetTypes} to redefine the types to try.
\classmember*{UncertainArrayHandle}{ResetTypes} has two template parameters that are the \vtkm{List}{}s of value and storage types.
\classmember*{UnknownArrayHandle}{ResetTypes} returns a new \textidentifier{UncertainArrayHandle} with the given types.
This is a convenient way to pass these types to functions.

\vtkmlisting{Resetting the types of an \textidentifier{UnknownArrayHandle}.}{UnknownArrayResetTypes.cxx}

\begin{commonerrors}
  Because it returns an \textidentifier{UncertainArrayHandle}, you need to include \vtkmheader{vtkm/cont}{UncertainArrayHandle.h} if you use \classmember{UnknownArrayHandle}{ResetTypes}.
  This is true even if you do not directly use the returned object.
\end{commonerrors}

\index{unknown array handle!cast|)}

\section{Accessing Truly Unknown Arrays}

So far in Sections \ref{sec:UnknownArrayHandle:CastToKnownType} and \ref{sec:UnknownArrayHandle:CastToPotentialTypes} we explored how to access the data in an \textidentifier{UnknownArrayHandle} when you actually know the array type or can narrow down the array type to some finite number of candidates.
But what happens if you cannot practically narrow down the types in the \textidentifier{UnknownArrayHandle}?
For this case, \textidentifier{UnknownArrayHandle} provides a mechanism for extracting data knowing little or nothing about the types by looking at a single \vtkm{Vec} component at a time.

You can use the \classmember{UnknownArrayHandle}{ExtractComponent} method to return an \textidentifier{ArrayHandle} with the values for a given component for each value in the array.
\classmember*{UnknownArrayHandle}{ExtractComponent} must be given a template argument for the base component type.
The following example extracts the first component of all \vtkm{Vec} values in an \textidentifier{UnknownArrayHandle} assuming that the component is of type \vtkm{FloatDefault} (line \ref{ex:UnknownArrayExtractComponent.cxx:Call}).

\vtkmlisting[ex:UnknownArrayExtractComponent.cxx]{Extracting the first component of every value in an \textidentifier{UnknownArrayHandle}.}{UnknownArrayExtractComponent.cxx}

The code in Example \ref{ex:UnknownArrayExtractComponent.cxx} works with any array with values based on the default floating point type.
If the \textidentifier{UnknownArrayHandle} has an array containing \vtkm{FloatDefault}, then the returned array has all the same values.
If the \textidentifier{UnknownArrayHandle} contains values of type \vtkm{Vec3f}, then each value in the returned array will be the first component of this array.

If the \textidentifier{UnknownArrayHandle} really contains an array with incompatible value types (such as \textidentifier{ArrayHandle}\tparams{\vtkm{Id}{}}), then an \vtkmcont{ErrorBadType} will be thrown.
To check if the \textidentifier{UnknownArrayHandle} contains an array of a compatible type, use the \classmember*{UnknownArrayHandle}{IsBaseComponentType} method.

\vtkmlisting{Checking the base component type in an \textidentifier{UnknownArrayHandle}.}{UnknownArrayBaseComponentType.cxx}

This section started with the motivation of getting data from an \textidentifier{UnknownArrayHandle} without knowing anything about the type, yet \classmember*{UnknownArrayHandle}{ExtractComponent} still requires a type parameter.
However, by limiting the type needed to the base component type, you only need to check the base C types (standard integers and floating points) available in C++.
You do not need to know whether these components are arranged in \textidentifier{Vec}s or the size of the \vtkm{Vec}.
A general implementation of an algorithm might have to deal with scalars as well as \textidentifier{Vec}s of size 2, 3, and 4.
If we consider operations on tensors, \textidentifier{Vec}s of size 6 and 9 can be common as well.
But when using \classmember*{UnknownArrayHandle}{ExtractComponent}, a single condition can handle any potential \textidentifier{Vec} size.

Another advantage of \classmember*{UnknownArrayHandle}{ExtractComponent} is that the type of storage does not need to be specified.
\classmember*{UnknownArrayHandle}{ExtractComponent} works with any type of \textidentifier{ArrayHandle} storage (with some caveats).
So, Example \ref{ex:UnknownArrayExtractComponent.cxx} works equally as well with \textidentifier{ArrayHandleBasic}, \textidentifier{ArrayHandleSOA}, \textidentifier{ArrayHandleUniformPointCoordinates}, \textidentifier{ArrayHandleCartesianProduct}, and many others.
Trying to capture all reasonable types of arrays could easily require hundreds of conditions, all of which and more can be captured with \classmember*{UnknownArrayHandle}{ExtractComponent} and the roughly 12 basic C data types.
In practice, you often only really have to worry about floating point components, which further reduces the cases down to (usually) 2.

\classmember{UnknownArrayHandle}{ExtractComponent} works by returning an \textidentifier{ArrayHandleStride}.
This is a special \textidentifier{ArrayHandle} that can access data buffers by skipping values at regular intervals.
This allows it to access data packed in different ways such as \textidentifier{ArrayHandleBasic}, \textidentifier{ArrayHandleSOA}, and many others.
That said, \textidentifier{ArrayHandleStride} is not magic, so if cannot directly access memory, some or all of it may be copied.
If you are attempting to use the array from \classmember*{UnknownArrayHandle}{ExtractComponent} as an output array, pass \vtkm{CopyFlag}{}\textcode{::}\classmember*{CopyFlag}{Off} as a second argument.
This will ensure that data are not copied so that any data written will go to the original array (or throw an exception if this cannot be done).

\begin{commonerrors}
  Although \classmember{UnknownArrayHandle}{ExtractComponent} will technically work with any \textidentifier{ArrayHandle} (of simple \textidentifier{Vec} types), it may require a very inefficient memory copy.
  Pay attention if \classmember*{UnknownArrayHandle}{ExtractComponent} issues a warning about an inefficient memory copy.
  This is likely a serious performance issue, and the data should be retrieved in a different way (or better yet stored in a different way).
\end{commonerrors}

Example \ref{ex:UnknownArrayExtractComponent.cxx} access the first component of each \textidentifier{Vec} in an array.
But in practice you usually want to operate on all components stored in the array.
A simple solution is to iterate over each component.

\vtkmlisting[UnknownArrayExtractComponentsMultiple.cxx]{Extracting each component from an \textidentifier{UnknownArrayHandle}.}{UnknownArrayExtractComponentsMultiple.cxx}

To ensure that the type of the extracted component is a basic C type, the \vtkm{Vec} values are ``flattened.''
That is, they are treated as if they are a single level \vtkm{Vec}.
For example, if you have a value type of \vtkm{Vec}{}\tparams{\vtkm{Id3}, 2}, \classmember*{UnknownArrayHandle}{ExtractComponent} treats this type as \vtkm{Vec}{}\tparams{\vtkm{Id}, 6}.
This allows you to extract the components as type \vtkm{Id} rather than having a special case for \vtkm{Id3}.

Although iterating over components works fine, it can be inconvenient.
An alternate mechanism is to use \classmember{UnknownArrayHandle}{ExtractArrayFromComponents} to get all the components at once.
\classmember*{UnknownArrayHandle}{ExtractArrayFromComponents} works like \classmember*{UnknownArrayHandle}{ExtractComponent} except that instead of returning an \textidentifier{ArrayHandleStride}, it returns a special \vtkmcont{ArrayHandleRecombineVec} that behaves like an \textidentifier{ArrayHandle} to reference all component arrays at once.

\vtkmlisting{Extracting all components from an \textidentifier{UnknownArrayHandle} at once.}{UnknownArrayExtractArrayFromComponents.cxx}

\begin{commonerrors}
  Although it has the same interface as other \textidentifier{ArrayHandle}s, \textidentifier{ArrayHandleRecombineVec} has a special value type that breaks some conventions.
  For example, when used in a worklet, the value type passed from this array to the worklet cannot be replicated.
  That is, you cannot create a temporary stack value of the same type.
\end{commonerrors}

Because you still need to specify a base component type, you will likely still need to check several types to safely extract data from an \textidentifier{UnknownArrayHandle} by component.
To do this automatically, you can use the \classmember*{UnknownArrayHandle}{CastAndCallWithExtractedArray}.
This method behaves similarly to \classmember*{UncertainArrayHandle}{CastAndCall} except that it internally uses \classmember*{UnknownArrayHandle}{ExtractArrayFromComponents}.

\vtkmlisting{Calling a functor for nearly any type of array stored in an \textidentifier{UnknownArrayHandle}.}{UnknownArrayCallWithExtractedArray.cxx}

\section{Mutability}

\index{unknown array handle!const|(}

One subtle feature of \textidentifier{UnknownArrayHandle} is that the class is, in principle, a pointer to an array pointer.
This means that the data in an \textidentifier{UnknownArrayHandle} is always mutable even if the class is declared \textcode{const}.
The upshot is that you can pass output arrays as constant \textidentifier{UnknownArrayHandle} references.

\vtkmlisting{Using a \textcode{const} \textidentifier{UnknownArrayHandle} for a function output.}{UnknownArrayConstOutput.cxx}

Although it seems strange, there is a good reason to allow output \textidentifier{UnknownArrayHandle}s to be \textcode{const}.
It allows a typed \textidentifier{ArrayHandle} to be used as the argument to the function.
In this case, the compiler will automatically convert the \textidentifier{ArrayHandle} to a \textidentifier{UnknownArrayHandle}.
When C++ creates objects like this, they can only be passed as constant references or by value.
So, declaring the output parameter as \textcode{const} \textidentifier{UnknownArrayHandle} allows it to be used for code like this.

\vtkmlisting{Passing an \textidentifier{ArrayHandle} as an output \textidentifier{UnknownArrayHandle}.}{UseUnknownArrayConstOutput.cxx}

Of course, you could also declare the output by value instead of by reference, but this has the same semantics with extra internal pointer management.

\begin{didyouknow}
  When possible, it is better to pass \textidentifier{UnknownArrayHandle}s as constant references (or by value) rather than a mutable reference, even if the array contents are going to be modified.
  This allows the function to support automatic conversion of output \textidentifier{ArrayHandle}s.
\end{didyouknow}

So if a constant \textidentifier{UnknownArrayHandle} can have its contents modified, what is the difference between a constant reference and a non-constant reference?
The difference is that the constant reference can change the array's content, but not the array itself.
If you want to do operations like doing a shallow copy or changing the underlying type of the array, a non-constant reference is needed.

\index{unknown array handle!const|)}

\index{array handle!unknown|)}
\index{unknown array handle|)}
