Enable writing fields with Vecs of unknown length

Like VTK, VTK-m has an explicit way to represent vectors/tuples in an array to express things such as 3D coordinates. Although the memory layout is (typically) the same between the two, there is a subtle but important limitation in the way VTK-m represents them.

In VTK, the size of a "tuple" is selected at runtime. This allows the size of the tuples to easily be adjusted to whatever is needed. When using an array with a tuple size determined at runtime, small arrays are allocated to move data in and out of the array.

However, this approach does not work well on a GPU because dynamically allocating memory of the appropriate size is problematic. Instead, VTK-m takes a slightly different approach. VTK-m defines a simple templated type named Vec that represents a short vector of components with a length defined at compile time. Because the length is defined at compile time, operating on arrays of indeterminate type is problematic. Operating directly on the data requires a separate compilation path for each vector length.

When the size of the Vec in a VTK-m array is not specifiable, then creating all code paths for any possible array type is problematic. Thus, VTK-m needs ways to operate on these arrays.

Fortunately, we have made some recent strides on making this possible for a variety of use cases, but some problems still exist. This issue will start with listing the features that are currently available and then list those that remain.

Features that are available

Several recent additions allow VTK-m algorithms to operate on arrays with arbitrary Vec lengths. The most important feature is the ArrayExtractComponent function that allows pulling a single component of the array out without copying data and without knowing the length of the Vecs. This is used for multiple methods of UnknownArrayHandle to extract arrays of known type regardless of what kind of array is stored.

It is also possible to create new arrays that have the same vector size as an input array. The NewInstance, NewInstanceBasic and NewInstanceFloatBasic methods of UnknownArrayHandle all provide a means of creating a new array with an unknown vector size although you need an existing array with the target vector size already.

It is also usually straightforward to create a scalar array with the same component type as that of an input array of Vecs. This is usually done after some sort of CastAndCall determines the component type of the input.

What is missing

There are some features that are still difficult or impossible with the current features.

Get extracted array with float fallback

As mentioned, UnknownArrayHandle has a way of extracting a component from an array knowing only the base component type of the array. This can be used to get an "extracted array," and this in turn is used with a special CastAndCall to get arrays of any storage or Vec length.

However, this CastAndCall does not have a means for a float fallback. This is less important than other CastAndCalls as there are only about 10 possibilities. However, for complex operations it might be better to just worry about a restricted set of types. Thus, it would be helpful to have a CastAndCallWithExtractedArrayWithFloatFallback. This mouthful of a method would allow someone to provide a vtkm::List of component types to try, and would convert to vtkm::FloatDefault if none of these worked.

Construct an array of arbitrary Vec length

Currently, to create an array of Vecs, you either need to know the size of the Vecs at compile time or have an array of the same Vec size you want. This can be problematic in some instances. For example, if a filter were to create a field that had Vecs with twice as many components as the input (say, computing a confidence interval), that could not be done in a single array. This also comes up a lot when importing data from other sources such as when reading data from a file or converting from VTK.

The proposed solution would be to create a new type of ArrayHandle that is a cross between ArrayHandleGroupVec and ArrayHandleGroupVecVariable. It would essentially behave like ArrayHandleGroupVecVariable in that the size of the Vecs could be assigned at runtime, but it would limit the array to having the same Vec size for every entry. In this respect it is like ArrayHandleGroupVec but would not have to set the size at compile time. We could call this class ArrayHandleGroupVecRuntime (unless a better name comes along).

We won't dwell on the implementation specifics of ArrayHandleGroupVecRuntime. The implementation will be very similar to ArrayHandleGroupVecVariable except that the offsets can be implied implicitly.

Of course, simply creating an ArrayHandleGroupVecRuntime does not really solve the problem. Sure, you could create this array and put it in a field, but we don't want to add yet another type of storage to compile into all the algorithms in VTK-m. However, we can get around this problem by doing some implicit conversions in UnknownArrayHandle.

UnknownArrayHandle already provides some implicit conversions when data does not need to be copied. For example, if you ask an UnknownArrayHandle for an ArrayHandleMultiplexer, then it will return the array if it matches any of the conditions of that array.

In the same way, we can create special checks for a basic ArrayHandle. When requesting an ArrayHandleBasic, UnknownArrayHandle can check if it contains an ArrayHandleGroupVecRuntime of the appropriate type and length. It should then be possible to share Buffer objects to transform the data to the expected type.

Although a bit more tricky, we should be able to do the opposite as well. That is, ask for an ArrayHandleGroupVecRuntime and have it match any ArrayHandleBasic of the appropriate component type.