\documentclass{article}

\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[cmex10]{amsmath}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{fancyvrb}
\usepackage{framed}
\usepackage{ifthen}
\usepackage{cite}
\usepackage[resetlabels]{multibib}
\usepackage{tabulary}
\usepackage{url}
\usepackage{xspace}
\usepackage{wrapfig}
\usepackage[pdfborder={0 0 0}]{hyperref}
\usepackage{verbatim}

\usepackage{color}
\definecolor{yellow}{rgb}{1,1,0}
\definecolor{black}{rgb}{0,0,0}
\definecolor{ltcyan}{rgb}{.75,1,1}
\definecolor{red}{rgb}{1,0,0}
\definecolor{gray}{rgb}{.6,.6,.6}
\definecolor{darkred}{rgb}{0.5,0,0}
\definecolor{darkgreen}{rgb}{0,0.5,0}

% Cite commands I use to abstract away the different ways to reference an
% entry in the bibliography (superscripts, numbers, dates, or author
% abbreviations).  \scite is a short cite that is used immediately after
% when the authors are mentioned.  \lcite is a full citation that is used
% anywhere.  Both should be used right next to the text being cited without
% any spacing. \hcite is a citation that I am hiding, perhaps because I am
% nearing the maximum number of citations for a journal.
\newcommand*{\lcite}[1]{~\cite{#1}}
\newcommand*{\scite}[1]{~\cite{#1}}
\newcommand*{\hcite}[1]{}

\newcommand{\etal}{et al.\xspace}

\newcommand*{\keyterm}[1]{\emph{#1}}

\newcites{mixedmode,predecessors,portability}{References,References,References}

\definecolor{shadecolor}{rgb}{0.98,0.98,0.82}

\makeatletter
\@ifpackageloaded{tex4ht}{
  \newenvironment{sidebar}[1]{
    \section*{#1\hfill{\tiny [sidebar]}}
  }{}
}{
  \newenvironment{sidebar}[1]{
    \vspace{8pt}
    \begin{shaded}
      \vspace{-12pt}
      \section*{#1\hfill{\tiny [sidebar]}}
  }{
    \end{shaded}
  }
}
\makeatother

\newcommand{\fix}[1]{{\color{red}\textsc{[#1]}}}
%\newcommand{\fix}[1]{}

% Avoid putting figures on their own page.
\renewcommand{\textfraction}{0.05}
\renewcommand{\topfraction}{0.95}
\renewcommand{\bottomfraction}{0.95}

% Make sure this is big enough so that only big figures end up on their own
% page but small enough so that if a figure does have to be on its own
% page, it won't push everything to the bottom because it's not big enough
% to have its own page.
\renewcommand{\floatpagefraction}{.75}

\begin{document}

\sloppy

% Currently just taken from poster. Will likely be changed based on actual
% contributions.
\author{Kenneth~Moreland \and Christopher~Sewell \and William~Usher \and
  Li-ta~Lo \and Jeremy~Meredith \and David~Pugmire \and James~Kress \and
  Hendrik~Schroots \and Kwan-Liu~Ma \and Hank~Childs \and Matthew~Larsen
  \and Chun-Ming~Chen \and Robert~Maynard \and Berk~Geveci}

\title{VTK-m: Accelerating the Visualization Toolkit for Massively Threaded
  Architectures}

\date{}

\maketitle

\section*{Abstract}

One of the most critical challenges for HPC scientific visualization is the
execution on massively threaded processors. Of the many fundamental changes
we are seeing in HPC systems, one of the most profound is a reliance on new
processor types optimized for execution bandwidth over latency hiding. Our
current production scientific visualization software is not designed for
these new types of architectures. To address this issue, researchers are
collaborating on building VTK-m, a framework that serves as a container for
algorithms, provides flexible data representation, and simplifies the
design of visualization algorithms on new and future computer architecture.

% Abstract should be no more than 150 words.

\vspace{1ex}
\noindent
\textbf{Keywords:} I.6.9.f Visualization systems and software; I.6.9.g
Visualization techniques and methodologies; G.1.0.g Parallel Algorithms;
B.2.1.b Algorithmic and Logic Structures, Parallel

\section{Introduction}

Although the basic architecture for high-performance computing platforms
has remained homogeneous and consistent for over a decade, revolutionary
changes are appearing on leading edge supercomputers, with plans for future
supercomputers containing even larger changes. One troubling attribute of
future high-performance computing (HPC) machines is the massive increase in
concurrency required to sustain peak computation; most project billions of
threads to achieve an exaflop\lcite{ScientificDiscoveryExascale2011}. This
increase is partially accredited to requiring more cores to achieve faster
aggregate computing rates\hcite{ExascaleRoadMap} and partially accredited
to using additional threads per core to hide memory
latency\hcite{ExascaleArchitecturesReport}. Due to limitations of cost and
power, the system memory will not commensurately increase, which signifies
algorithms will need good strong scaling (that is, more parallelism per
datum)\hcite{DARPAExascaleStudy}.

This trend to massive threading can be seen in high-performance computing
today. The current leadership-class computer at Oak Ridge National
Laboratory, Titan, requires between 70 and 500 million threads to run at
peak, which is over 300$\times$ more than required by its predecessor,
JaguarPF. In contrast, the system memory grew only by a factor of $2.3$.

The increasing reliance on concurrency to achieve faster execution rates
invalidates the scalability of much of our scientific HPC code. New
processor architectures are leading to new programming models and new
algorithmic approaches. The design of new algorithms and their practical
implementation are a critical extreme-scale
challenge\lcite{ASCACSummaryReport2010,Childs2013}.

To address the needs for HPC scientific visualization, researchers working
for the United States Department of Energy are building a new library,
named VTK-m. VTK-m provides a framework for simplifying the design of
visualization algorithms on current and future architecture. VTK-m also
provides a flexible data model that can adapt to many scientific data types
and operate well on multithreaded devices. Finally, VTK-m serves as a
container for algorithms designed in the framework and gives the
visualization community a common point to collaborate, contribute, and
leverage massively threaded algorithms.

\begin{sidebar}{Scientific Visualization Moves to Mixed Mode Parallelism}

\renewcommand*{\lcite}[1]{~\citemixedmode{#1:mixedmode}}
\renewcommand*{\scite}[1]{~\citemixedmode{#1:mixedmode}}

For many years HPC visualization applications such as ParaView, VisIt,
EnSight, and FieldView have supported parallel processing on distributed
memory computer systems. The approach of all these software products is a
bulk synchronous parallel model where algorithms perform the majority of
their computation on independent local operations\lcite{Law1999}.

This model of parallel computation has worked well for the last 15 years.
Even recent multicore processors could be leveraged reasonably efficiently
as independent message-passing processes on each core, allowing these tools
to scale to petascale machines\lcite{Childs2010}.

However, processors designed for HPC installations are undergoing
transformative design changes. With physical limitations that prevent
individual cores from executing instructions appreciably faster than their
current rate, manufacturers are increasing the total computational
bandwidth by adding more cores to each processor\lcite{Fuller2011}. Some
HPC processor designs go even further to increase the total possible
execution throughput by removing latency-hiding features and incorporating
vector processing. A consequence of all these features is that it is no
longer sufficient to treat each core as an independent processor.

The upshot is that our parallel computing model is no longer symmetric. The
relationship between two cores on the same processor is very different than
the relationship between two nodes of a supercomputer. To address this
asymmetry, a popular approach is to use a mixed mode, or ``hybrid,''
parallel computing model that incorporates two levels of task organization.
The first level comprises distributed-memory message-passing nodes in a
cluster-like arrangement.
%% The de facto implementation for this model is MPI.
Then, within each node of the distributed-memory arrangement is a
processor or small set of processors capable of executing numerous threads
that may require coordination.
%% The implementation of this threaded
%% parallelism can vary depending on the underlying hardware and needs of the
%% software, but CUDA and OpenMP are popular choices.

The principle advantage of the mixed mode parallel model from our point of
view is that we can leverage our existing software to manage the message
passing parallel units. The VTK-m framework discussed in this paper focuses
on the intra-node parallelism that often requires massive threading and
synchronized execution.

\bibliographystylemixedmode{unsrt}
\bibliographymixedmode{VTKmCGA2015}

\renewcommand*{\lcite}[1]{~\cite{#1}}
\renewcommand*{\scite}[1]{~\cite{#1}}

\end{sidebar}

\section{The Challenges of Highly Threaded Visualization}

The scientific visualization research community has been building scalable
HPC algorithms for over 15 years, and today there are multiple production
tools that provide excellent scalability. However, our current visualization
tools are based on a message-passing programming model. They expect a
coarse decomposition of the data that works best when each processing
element has on the order of a hundred thousand to a million data cells.

HPC systems are undergoing a revolution in processor hardware architecture
(see the ``Scientific Visualization Moves to Mixed Mode Parallelism''
sidebar). New HPC systems require a much higher degree of parallelism that
could require threads operating on as few as one to ten data cells. At this
fine degree of parallelism, our conventional visualization breaks down in
multiple different ways.

\subsection{Load Imbalance}

Typically data are partitioned under the assumption that the amount of work
per datum is uniform. However, this is not true for all visualization
algorithms, many of which generate data conditionally based on the input
values. With only a few exceptions, current parallel visualization
functions completely ignore this load imbalance, which is considered
tolerable when amortized over larger partitions.

When the data gets decomposed to the cell level, this amortization no
longer occurs, which results in a much more severe load imbalance. Finely
threaded visualization algorithms need to be cognizant of potential load
imbalance and schedule work accordingly.
% Citation on reverse index lookup?

\subsection{Dynamic Memory Allocation}

When the amount of data a visualization algorithm generates is dependent on
the values of the input data, the size and structure of the output data is
not known at the execution outset. In such a case, the algorithm must
dynamically allocate memory as data are generated.

Because our conventional parallel visualization algorithms operate on
coarse partitions in distributed memory spaces, processing elements can
dynamically allocate memory completely independent from one another. In
contrast, dynamic memory allocation from many threads within a shared
memory environment requires explicit synchronization that inhibits parallel
execution.

%% Thus, dynamic memory allocation in a finely threaded algorithm requires a
%% different two-phased approach. The first parallel phase counts the output
%% data without generating it. This count information is used to allocate
%% memory between the phases. The second parallel phase generates the data and
%% stores them in a known region of the memory allocation.
% Citation on stream compaction?

\subsection{Topological Connections}

Scientific visualization algorithms are dominated by operations on
topological connections in meshes. Care must be taken when defining these
connections across boundaries of data assigned to different processing
elements. Mutual data being read must be consistent and mutual data being
written must be coordinated.

A conventional parallel visualization algorithm typically manages
topological connections across processes by replicating data on the
boundaries. Since the partitions are coarse, there is a limited amount of
replication. However, this replication cannot be sustained as the data
decomposition approaches single cells. Instead, threads must share
connected data, and generated data may require further processing to
identify coincident topology.
% Citation on geometry generation?

\subsection{SIMD Execution}

In addition to increasing execution bandwidth through multiple processing
cores, modern HPC processors also enable vector processing, which allows
the same instruction to be executed simultaneously on multiple data values.
Whereas conventional parallel visualization algorithms allow for completely
independent execution, efficient vector processing requires at least some
parallel execution to run in single instruction, multiple data (SIMD) mode.
New algorithms should minimize code execution divergence.

\begin{sidebar}{Predecessors of VTK-m}

\renewcommand*{\lcite}[1]{~\citepredecessors{#1:predecessors}}
\renewcommand*{\scite}[1]{~\citepredecessors{#1:predecessors}}

Although the VTK-m software project itself started little more than a year
ago, the software originated as an aggregation of three predecessor products:
PISTON, Dax, and EAVL. The US DOE HPC community predicted early on that
leadership class facilities would be transitioning to heavily threaded
processors and that we would need a significant change to our visualization
algorithms and software\lcite{ScientificDiscoveryExascale2011}.
Consequently, researchers at the DOE national laboratories began
considering the challenges of visualization on accelerator processors and
three separate toolkits, each focusing on a specific aspect of the problem,
were created.

The first toolkit, PISTON\lcite{Lo2012}, considers the design of portable
multithreaded visualization algorithms. Built on top of the Thrust
library\lcite{Thrust}, algorithms in PISTON comprise a sequence of general
parallel operations. Originally designed for CUDA, Thrust has a
flexible device back-end that now supports multicore CPU architectures.
PISTON has demonstrated scalability across multiple devices.

The second toolkit, Dax\lcite{Moreland2012:PDAC}, considers a top-down
approach to a multithreaded visualization framework. Recognizing that most
visualization algorithms iteratively apply an operation to each element in
a mesh, the Dax framework allows developers to concentrate on designing
these per-element operations while the framework automatically builds the
appropriate parallel scheduling.

The third toolkit, EAVL\lcite{EAVL2012EGPGV}, considers the data model used
in modern visualization. EAVL updates the visualization data model to
handle modern simulation codes using flexible mesh structures that are also
easily accessible in multithreaded environments.

Each originally focused on a different part of the extreme
scale visualization problem, but the software packages did not integrate
well. Recognizing the prospect of substantial duplication of effort, the
developers of these software projects came together to work under a unified
software product: VTK-m. Although VTK-m is born from a new code base, the
developers of PISTON, Dax, and EAVL have contributed and evolved their
respective technology. While development of its predecessors has been
phased out, VTK-m is now a unified, well-integrated product of these three
predecessors with continuing evolving capabilities.

\bibliographystylepredecessors{unsrt}
\bibliographypredecessors{VTKmCGA2015}

\renewcommand*{\lcite}[1]{~\cite{#1}}
\renewcommand*{\scite}[1]{~\cite{#1}}

\end{sidebar}

\section{Visualization with Data Parallel Primitives}

Data parallel primitives provide an abstraction layer between the low-level
hardware architecture and the high-level code, which both increases the
portability of VTK-m (see the ``Performance Portability with Data Parallel
Primitives'' sidebar) and simplifies the implementation of algorithms.
Blelloch\lcite{Blelloch1990} originally proposed a scan-vector model for
data-parallel computing, and outlined a wide variety of algorithms in data
structures, computational geometry, graphs, and numerical analysis. All of
VTK-m's predecessors (see the ``Predecessors of VTK-m'' sidebar) leverage
data parallel primitives in some way. VTK-m provides an abstract device
model inherited from its predecessor Dax and expanded to fulfill the needs
of the algorithms inherited from its predecessor PISTON. VTK-m makes use of
the following data parallel primitives.

\begin{description}
\item[Map] Applies the same unary operator to each element of an array in
  parallel. This is useful for element-wise field expressions or can be
  leveraged as a general scheduling capability.
\item[Reduce] Computes a single value (such as the sum or product) from all
  the elements of an array, which is useful whenever aggregate values are
  needed.
  %% A reduction can be implemented in a logarithmic
  %% number of parallel steps using a standard tree-reduction algorithm.
  %% Generation primitives can allocate and initialize vectors, optionally as
  %% ``virtual'' vectors that require no actual storage if the initialization
  %% values can be computed analytically (such as a vector of a constant
  %% value, or of values that increase by a constant increment). These
  %% primitives can be combined to accomplish a wide range of operations. For
  %% example, a simple 2D numerical quadrature (midpoint rule) can be computed
  %% by generating a (virtual) vector of x-values with a constant increment,
  %% using a transform to evaluate a function at each of these x-values,
  %% another transform to multiply these function values by $\Delta x$, and
  %% finally a reduction to sum the areas of all of the rectangles.
\item[Scan] Similar to a reduction but stores intermediate results (that is, a
  ``running total'') at each element of the array. Scans are particularly
  useful for finding array indices.
  %% Scan can be computed
  %% using a two-pass algorithm (each with a logarithmic number of parallel
  %% steps) by first computing partial sums similar to the tree-reduction
  %% algorithm in the first ``up-sweep'' pass and then filling in additional
  %% intermediate values on a second ``down-sweep'' pass. By using custom
  %% binary operators, rather than just the addition operator, and custom data
  %% types, much more than partial sums can be computed. For example,
  %% generating the Fibonacci sequence, which might appear to be an inherently
  %% sequential computation, can be computed in parallel by a scan on a vector
  %% of 2x2 matrices with a matrix multiplication binary operator.
\item[Sort] A versatile reordering operation that is particularly useful
  for identifying all groups of common values in an array.
\item[Search] Can simultaneously find multiple values in an array. This can
  be used to find a reverse mapping of values in one input array to their
  locations in a second sorted input array.
  %% This reverse map is often
  %% useful in algorithms with sparse output by first using a map to identify
  %% how much output each input element generates, then second using a scan to
  %% identify a unique output location, and finally using a vectorized search
  %% to identify for each output where each input comes from.
\end{description}

% This seems to be the same description of search put in a different way.
%Scatters and gathers are useful for performing index computations.  For example, the sorted rank of each value in a vector can be determined by sorting the vector of values along with their indexes, and then scattering their original indexes by the sorted indexes.  The dynamic memory allocation issue described above can be addressed for the isosurface example by gathering the number of isosurface vertexes generated by each geometry-producing cell using the ``reverse map'' indexes.  A scan on the resulting vector then yields the offset into the global output vector at which each cell should write its geometry.

Algorithms that employ hierarchical parallelism may make use of
``segmented'' versions of data parallel primitives. For example, when
constructing KD-trees or other types of graphs, the structure of the graph
may be encoded using a segment vector, where each segment represents a
node. Operations such as sorts and scans can then be performed
independently within each node but still all in parallel.

Although research has found effective ways to utilize these data parallel
primitives to many visualization problems, the application of these
primitives is not always obvious. Fortunately, many visualization
algorithms follow similar patterns of
operation\lcite{Moreland2013:UltraVis}, and these patterns are embodied by
objects called worklets in VTK-m, a concept inherited from the predecessor
Dax.

Worklets extend the functionality of the basic data parallel primitives by
providing common features. For example, several flavors of worklets can
trace topological connections in map operations. The indexing of the
topology structure is handled internally by VTK-m, which makes it easier
and safer to write worklets than to directly operate on data structures in
parallel. Other worklet types can manage algorithms with sparse output or
find coincident topology, further reducing the burden of the programmer.

%% Much of the power of data parallel primitives comes from the ability of the
%% programmer to customize their behavior by implementing their own functors
%% or worklets. However, care must be taken when implementing these operators
%% to avoid compromising the correctness or performance of the algorithm. As
%% with functional programming languages, the safe parallel scheduling of
%% these operators requires that they not have side effects; that is, they do
%% not directly modify global data. Furthermore, although primitives such as
%% scan and reduce can be implemented to run very efficiently for simple
%% arithmetic operators, complex user-defined worklets with significant
%% branching or inefficient memory accesses can greatly compromise
%% performance.

\begin{sidebar}{Performance Portability with Data Parallel Primitives}

\renewcommand*{\lcite}[1]{~\citeportability{#1:portability}}
\renewcommand*{\scite}[1]{~\citeportability{#1:portability}}

Unlike many ``in house'' HPC software applications that can be written for
a specific HPC installation, the software our team develops is used across
all the US Department of Energy supercomputer facilities and many others
around the world. This means our software must perform well on a variety of
hardware types and configurations. Furthermore, the critical code in a visualization
system is not limited to a small number of iterative loops. Our software
contains numerous algorithm implementations to address visualization needs
across many scientific disciplines. For example, the Visualization Toolkit,
on which ParaView and VisIt are based, contains over 400
distinct filters.

Upgrading this functionality is a daunting challenge. Attempting to create
multiple algorithm implementations targeted to each known platform (and
possibly currently unknown platforms in the future) creates a combinatorial
explosion of work. This approach would require significant developer
investment, likely exceeding worldwide funding available for this task.

Instead, VTK-m uses data parallel primitives to achieve performance
portability. VTK-m defines an abstract device model constructed from a set
of well-known operations such as map, scan, sort, and reduce, that
run in parallel on the device. The VTK-m team is rethinking visualization
algorithms as a sequence of these parallel primitive operations. When
implemented as such, the entirety of algorithms in VTK-m can be ported to a
new device by providing only the parallel primitives for that device.
%% The
%% implementation of these parallel primitives can be complex, but it is a
%% small, localized section of code.
Algorithms need only be written once to VTK-m's abstract device model,
which dramatically reduces the implementation work for various
architectures as shown in Figure~\ref{fig:MxN}.

% As before, use minipage instead of figure. See the comment above.
\noindent
\begin{minipage}{\textwidth}
  \centering
  \includegraphics{images/MxN_Problem}%
  \includegraphics{images/MxN_Solution}
  \captionof{figure}{Implementing all of our visualization algorithms
    separately for every architecture (at left) we need to support leads to
    an unmanageable amount of software. By using VTK-m's abstract device
    model based on data parallel primitives (at right), we can reduce the
    amount of software to implement to a feasible level.}
  \label{fig:MxN}
\end{minipage}
\vspace{\baselineskip}

The indirection of using data parallel primitives to implement algorithms
can limit the amount of optimization that can be done, which raises the
question of how much performance portability costs. Evidence collected from
recent research using data parallel primitives\lcite{Lo2012,Larsen2015}
including work presented in this paper suggests a fairly low overhead. The
benefits of simplified implementation and performance portability outweigh
any overheads introduced by the approach.

\bibliographystyleportability{unsrt}
\bibliographyportability{VTKmCGA2015}

\renewcommand*{\lcite}[1]{~\cite{#1}}
\renewcommand*{\scite}[1]{~\cite{#1}}

\end{sidebar}

\section{Advanced Data Models}

VTK-m is intended to be used in many scientific disciplines and with data
of many different arrangement and structure. It is important that the data
model in VTK-m be flexible enough to capture the different representations
of data while still providing clear semantics for the elements in the data.
Furthermore, the data representation must be space efficient and be
accessible on the different processor types we use (that is, work on both
CPU and GPU). To achieve these goals, VTK-m provides a data model inherited
from its predecessor EAVL (see the ``Predecessors of VTK-m'' sidebar) with
some modifications and extensions.

The utility of the data models is further enhanced by integrating them with
the aforementioned data parallel primitives, abstract device adapters, and
worklet mechanisms. Data sets in VTK-m seamlessly move between devices.
Algorithms built within VTK-m use abstract concepts that can be applied to
data sets of many different structures without having to be reimplemented.

\subsection{Data Sets}

Traditional data set models often choose a set of
rigid characteristics for a data set, and then label
that collection of characteristics as a specific type of mesh.
For example, a uniform data set has regular
axis-aligned coordinates and a logical
$[i, j, k]$ cell arrangement. An unstructured data set has
fully explicit coordinates (a $[x, y, z]$ value separately
defined for each point) with fully explicit cell connectivity
defined by arrays of indices.
A curvilinear grid is a mix with logical cell arrangement but explicit
coordinates.

However, this selection of grid types is somewhat arbitrary. It is easy to
generate unsupported meshes. For example, consider a tetrahedralization of
a uniform grid. This would be most efficiently represented with regular
coordinates and explicit cells, but most data models require a fully
explicit unstructured grid.

%% In some sense, these two typical examples can be viewed as
%% extremes in a spectrum of explicitness. The uniform data is
%% fully implicit whereas the unstructured data is fully explicit.
%% Some hybrids are common: What is generally called
%% a curvilinear grid has explicit
%% coordinates with a logical cell arrangement.  However,
%% the selection of hybrids in most data models is somewhat arbitrary; it matches
%% what may be common mesh selections for simulation codes,
%% but it is easy to generate unsupported meshes
%% during visualization and analysis operations.
%% For example, imagine using a
%% threshold operation to remove cells from a uniform
%% grid; point coordinates can remain efficiently defined
%% with implicit spacing, but you no longer have a
%% fully logical set of cells defined across these points.
%% In cases like this, one must fall back to the less
%% efficient, fully explicit, unstructured grid to represent
%% their data, in the process losing out on any more
%% computationally efficient algorithms that might apply
%% to their grid if a more exact representation were available.

VTK-m removes this restriction entirely by defining a data set as a mere
container for arbitrary collections of coordinates, cell sets, and fields.
Although this allows VTK-m to support more mesh types, the algorithm
designer does not have to manage this complexity. At the worklet level cell
shape and field values are decoupled from the mesh representation, meaning
the algorithm need only be implemented once for all mesh types.

%% This additional flexibility might initially appear to increase the burden
%% for algorithm developers, as they now must support an extremely high number
%% of mesh types when taking into consideration the total combinations
%% available to data set creators. However, one of our key insights is that
%% when operating on data sets at the cell level in a worklet, the computation
%% is best described using the shape of the cell and the value of fields at
%% the cell nodes. Cell shape and field values can be decoupled from the
%% representation of the cell connectivity and coordinates system (which is
%% really just a special field). Thus, the additional flexibility is actually
%% a simplification as algorithm developers need not be burdened by the data
%% representation.

The ability to correctly represent mesh structures
is important for computational efficiency and accuracy.
It is also critical for in situ visualization and analysis
tasks, as any mismatch between the representational
capabilities of the analysis code relative to the targeted
simulation code will result in expensive (both computationally
and in terms of memory footprint) copies and transformations
of data arrays.

\subsection{Coordinate Systems}

Because it allows an arbitrary collection
of coordinates, VTK-m supports data sets with no
coordinates (such as graph data with no physical location
for the vertices) or data sets with coordinate systems of spatial
dimension other than three.  VTK-m also supports more than
one coordinate system, which recognizes that a mesh might
be interpreted in more than one spatial mapping
independent of the underlying mesh structure.
For example, geospatial data could simultaneously have coordinate systems
defined by 3D position, latitude-longitude coordinates, and any number of
2D projections.

%% The fundamental array primitives in VTK-m then add
%% additional flexibility, such as supporting both strided
%% and interleaved coordinate arrays.  This is also the
%% mechanism to support coordinate fields in logical dimensions
%% such as rectilinear or even a mixture where some
%% coordinates are implicit and others are explicit.
%% This combination of coordinate types is difficult to represent efficiently
%% in a traditional data model but occurs in practice. For example, an implicit
%% two-dimensional uniform grid with an explicit $z$ elevation is useful for
%% representing elevation maps, terrains, and carpet plots.

\subsection{Cell Sets}

A common problem in traditional data set models
is the inability to properly support more than one
type of topological element in the same mesh,
such as volumetric elements with boundary interface polygons
or point elements with connecting line segments.
Although it can often be supported by merging these
disparate topological element types into a single
mesh, the lack of separation requires that fields
contain dummy values for the element types for
which they are inapplicable, an error-prone and
inefficient practice.

VTK-m solves this problem by allowing multiple
cell sets to be contained in the same mesh. The
unifying characteristic of belonging to the same
mesh (as opposed to two different meshes) is that
they both refer to the same set of points using
the same set of indices, which allows efficient
mapping between these two cell sets.

%% The topological mapping worklets operate on one
%% of these cell sets at a time, as they define the
%% mapping between topological elements.
%% For logically structured
%% grids, the connectivity for any pair of topological
%% elements (such as points and lines) can be derived
%% programmatically from knowing only the $[i, j, k]$ dimensions
%% for the grid.  For fully explicit grids,
%% the connectivity is either passed in or derived
%% from another connectivity type; for example, users
%% typically will define an explicit cell set using
%% a point-to-cell mapping, and VTK-m can use this to derive
%% other mappings (face-to-edge, cell-to-point, etc.).
%% For a hybrid structured/explicit cell set, such as
%% one which is an irregular triangle mesh in the plane
%% but is extruded along a third dimension, the connectivity
%% is a combination of the logical dimension and
%% explicit point identifiers.

\section{Use Cases}

There has been significant interest within the scientific visualization community to experiment
with and adopt the VTK-m framework. This section contains several examples of VTK-m being used
in a variety different projects, including volume rendering, ray tracing, geometric operations, and use in workflows for large scale scientific applications.

\subsection{Ray Tracing}

Larsen \etal\lcite{Larsen2015} implement a ray tracer using data parallel primitives
in an effort to provide a many-core capable infrastructure that can perform
rendering on any architecture.
%
Ray tracing is a compelling algorithm for
exploring the boundaries of data-parallel primitives
because it is computationally intensive and has unstructured memory accesses.
%
Furthermore, the HPC community has a growing interest in using ray tracing
rather than the traditional rasterization-based rendering as the
computational complexity of ray tracing is much better with respect to the
size of the geometry (which has been steadily growing in the HPC setting).

\begin{figure}[htb]
  \centering
  \includegraphics[width=.5\linewidth]{images/RayCastingSeismic}
  \caption{Example output from VTK-m ray tracing algorithm.}
  % Image is of ``Seismic'' data set, courtesy of Oak Ridge National
  % Laboratory representing seismic wave propagation through the Earth as
  % generated by the SPECFEM3D simulation.
  \label{fig:RayCasting}
\end{figure}

\begin{table}[htb]
  \caption{The speed (measured in millions of rays cast per second) to
    perform ray tracing using different algorithms on different data sets.
    At left are times for a GPU. At right are times for a CPU.}
  \label{tab:RayCastingTimes}
  \centering
  \includegraphics[width=.49\linewidth, trim=0.505in 0.517in 0.505in 0.505in]{data/RayCastingTimesGPU}
  \includegraphics[width=.49\linewidth, trim=0.505in 0.517in 0.505in 0.505in]{data/RayCastingTimesCPU}
\end{table}

To verify the efficiency of the VTK-m framework, we have implemented
Larsen's ray tracing algorithm in VTK-m and replicated many of the
experimental runs in Larsen's paper\lcite{Larsen2015}. Example output from
the ray tracer is shown in Figure~\ref{fig:RayCasting}, and the results of
our experiments are summarized in Table~\ref{tab:RayCastingTimes}. The data
sets are the same ones used in Larsen's study, and the timings for the
Optix Prime, Embree, and EAVL algorithms are taken from Larsen's paper. The
GPU runs come from a Tesla K40 and the CPU runs come from dual Intel Xeon
E5's, which are equivalent to the GPU2 and CPU2 configurations,
respectively, in Larsen's paper. Likewise, all images are rendered with
1080P resolution ($1920\times1080$) as was reported by Larsen.

Our experiments verify the conclusions drawn by Larsen's study in that our
architecture-agnostic data-parallel approach is competitive to (within a
factor of 2) highly optimized architecture-specific approaches. Also, the
VTK-m implementation is at least as fast as, and sometimes faster than,
Larsen's previous implementation written using EAVL (one of VTK-m's
predecessors), which suggests that the integration of data models with the
execution model, algorithms, and other features of VTK-m does not add
overhead.

%% Larsen's code is a functional ray tracer assembled from data parallel primitives.
%% %
%% It is capable of both advanced lighting effects and
%% simplified lighting that matches rasterization.
%% %
%% The code was initially implemented in EAVL, but is currently being migrated to VTK-m.
%% %
%% Its usage of data parallel primitives (primarily map operations) makes
%% efficient use of parallel hardware via device implementations of the primitives,
%% and no architecture-specific code is used.

%% Larsen's study looks at performance on CPUs, GPUs, and the Intel Xeon Phi with
%% a variety of ray-tracing workloads.
%% %
%% The execution times for all configurations are
%% consistent with interactivity constraints.
%% %
%% Furthermore, the study compares each experiment to NVIDIA's OptiX Prime and Intel's Embree,
%% ray-tracing packages designed and optimized specifically for their respective architectures.
%% %
%% Performance is consistently within a factor of two and even outperforms the packages
%% in some cases.
%% %(specifically on older NVIDIA GPUs, since OptiX's focus is on the newest
%% %NVIDIA cards).
%% %
%% Overall, this study provides encouragement that the data-parallel approach is effective
%% as it demonstrates that an architecture-agnostic approach can
%% be competitive with architecture-specific approaches.

\subsection{Direct Volume Rendering}
%Ray casting is a volume rendering technique used to rasterize 3D data sets. To accomplish this a ray is traced from a center of projection though each pixel in the viewing window. Each ray that intersects the volume samples the volume at regular intervals. The value at each sample point is mapped to a transfer function which identifies what color that sample point should be. As the ray travels through the volume the color is accumulated. This method is a discrete approximation of the volume rendering integral.\fix{Cite}
%The nature of the algorithm allows each ray to be traced independently, which makes ray casting a parallel algorithm that can be expressed using data parallel primitives.

Direct volume rendering, in which we render a volume in space with variable
color and opacity, is a common but computationally intensive task in
scientific visualization. Thus, direct volume rendering is a good use case
to demonstrate the capabilities of VTK-m. We have implemented a very simple
ray-casting direct volume renderer within the VTK-m framework. An example
output of this volume renderer is shown in
Figure~\ref{fig:SuperNovaPhongShading}.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.5\linewidth]{images/Entropy_compressed}
  \caption{Rendered image of supernova simulation data using a direct
    volume renderer implemented in VTK-m.}
  \label{fig:SuperNovaPhongShading}
\end{figure}

%% To demonstrate the usability of VTK-m we implement a volume ray caster and
%% compare it to a ray caster using Dax, a predecessor to VTK-m. The two
%% versions are nearly identical using a simple map-based worklet in which
%% each instance casts a ray through a pixel. Using the portability of each
%% framework to use different back ends, we compile the same user-level code
%% for three different architectures. Each architecture is representative of
%% technology used in today's supercomputers.
%% \begin{description}
%% \item[GPU] NVIDIA Titan X, 2,880 stream processors, 6~GB on chip memory,
%%   889~MHz
%% \item[CPU] 2x Intel Xeon E5-2620, 6 cores per processor and 2x
%%   hyperthreading per core, 64~GB system memory, 2.2~GHz
%% \item[MIC] Intel Xeon Phi 5110p many integrated cores (MIC) co-processor,
%%   60 cores with 4x hyperthreading per core, 8~GB on chip memory, 1.1~GHz
%% \end{description}

%Each ray samples the volume 512 times along its path. The data values are normalized in a pre-computation step and are not included in the timings.
For testing, the output of a supernova simulation data set is used,
consisting of a $432^{3}$ volume of voxels, and rendered to a $1024 \times
1024$ window. Because performance is directly related to the number of rays
that intersect the volume, and ultimately generate a color, the camera is
positioned so that the volume occupies a large portion of the viewing
window. The rendering time is averaged over the rendering of 16 frames.

We test the same VTK-m volume rendering on three different devices:
dual-processor Intel Xeon IvyBridge E5-2670 v2 (labeled CPU), an NVIDIA
Titan X (labeled GPU), and an Intel Xeon Phi 5110p co-processor (labeled
MIC). For comparison purposes, we also render the volume using VTK (with
the \texttt{vtkFixedPointVolumeRayCastMapper} class). This VTK
implementation is multithreaded but designed strictly for the CPU so only
run on that architecture. We also compare our VTK-m volume rendering
algorithm with a very similar one implemented in Dax, one of VTK-m's
predecessors. The results of these experiments are summarized in
Table~\ref{tbl:RayCasterRuntime}. All implementations are multithreaded and
run on all cores available.

%% The GPU
%% experiments use the CUDA device whereas the CPU and MIC experiments use the
%% Intel Threading Building Blocks (TBB) device.

%% An important factor in volume rendering is real time interactivity. Real
%% time response allows users to gain information about the volume using parallax,
%% which is the observation of changing depth cues in the image due to rotations.
%% The data in Table \ref{tbl:RayCasterRuntime} shows that the GPU device
%% approaches interactive frame rates for the VTK-m version. Across all three
%% architectures we see an improvement in run time for ray casting using
%% VTK-m's new data model and scheduling mechanics.
%The CUDA version run on \textbf{GPU} improves from $3fps$ to about $6.8fps$ on average. \textbf{CPU}, and \textbf{MIC} see even larger performance gains going from approximately $0.2fps$ to $0.5fps$ and $0.07fps$ to $0.3fps$ respectively on average.

%\begin{figure}[htb]
%	\centering
%		\includegraphics[width=0.5\linewidth]{images/RayCasterRuntime}
%		\caption{Average time, in seconds, to render a single frame for all three architectures. The back end used is included between the parentheses.}
%		\label{fig:RayCasterRuntime}
%\end{figure}

%%
% Full resolution timing data (original)
%      Arch                 | Dax RC (sec) | VTKm RC (sec)| rel speedup
%    ------------------------------------------------------------------
%    NVIDIA Titan X (CUDA)  | 0.281841569  |  0.147178    | 1.9x
%    Intel Xeon (TBB)       | 5.379840625  |  2.065675333 | 2.6x
%    Intel Xeon Phi (TBB)   | 13.82085     | 3.587642835  | 3.9x
%%
\begin{table}[htb]
  \centering
  \caption{Average rendering rate for a 1024 x 1024 volume rendering image.}
  \label{tbl:RayCasterRuntime}
  \includegraphics[width=\linewidth, trim=0.505in 0.517in 0.505in 0.505in]{data/VolumeRenderingTimes}
  %% \begin{tabular}{lrrr}
  %%   & \multicolumn{1}{c}{Dax} & \multicolumn{1}{c}{VTK-m} & \multicolumn{1}{c}{Speedup} \\
  %%   \midrule
  %%   GPU & 3.55 fps & 6.80 fps & 1.9$\times$ \\
  %%   CPU & 0.19 fps & 0.48 fps & 2.6$\times$ \\
  %%   MIC & 0.07 fps & 0.28 fps & 3.9$\times$ \\
  %% \end{tabular}
\end{table}

The comparison between our VTK-m volume rendering and the existing VTK
volume rendering is not a direct comparison of the performances of the two
respective libraries as there are significant differences between the two
algorithm implementations. In particular, the VTK-m volume rendering is
much simpler than the VTK volume rendering, which has optimizations such as
empty space skipping not yet implemented in our algorithm. Nevertheless,
our simpler algorithm performs comparatively well and is also portable
across multiple architectures.

In contrast, the volume rendering implementations in VTK-m and Dax are very
similar. We can clearly see that the integration of the Dax technologies
with those of the other VTK-m predecessors has not resulted in a larger
overhead. On the contrary, the VTK-m framework is operating more
efficiently.

%% Another measurement to characterize the behavior of an application is the
%% number of executed instructions per cycle (IPC). Since data-intensive
%% applications will often have to stall instructions to wait for data to
%% load, IPC is a good indicator of the algorithm's performance. Additionally,
%% the NVIDIA profiler reports the fraction of instruction stalls due to data
%% load requests, which helps diagnose imperfect IPC measurements. The number
%% of registers used per thread also affects NVIDIA GPUs. Fewer registers used
%% per thread means more threads can be used on a single core. More threads on
%% a core provides better memory latency hiding. The VTK-m version shows a
%% 23.2\% reduction in registers used, increasing parallelism and overall
%% throughput. As Table \ref{tbl:GPUMetrics} shows, the reduced percentage of
%% instructions stalled due to data requests, increased IPC, and increased
%% occupancy all demonstrate better data performance by VTK-m.

%% \begin{center}
%% \begin{table}[htb]
%%   \centering
%%   \caption{Registers per thread, IPC, stalls due to data request, and
%%     occupancy compared between the Dax and VTK-m ray casters for GPU.}
%%   \label{tbl:GPUMetrics}
%%   \begin{tabular}{lrrrr}
%%     & Registers & \multicolumn{1}{c}{IPC} & Data Load Stalls & Occupancy \\
%%     \midrule
%%     Dax & 95 & 0.606 & 50.8\%  & 28.7\% \\
%%     VTK-m & 73 & 0.726 & 45.4\% & 30.2\% \\
%%   \end{tabular}
%% \end{table}
%% \end{center}

\subsection{Isosurface}

Isosurfacing is one of the most valuable and widely used algorithms in
scientific visualization. Our isosurface algorithm is adapted from PISTON,
a predecessor to VTK-m. The new implementation integrates the algorithm
with VTK-m's data model and worklet execution mechanism.

We demonstrate the isosurface algorithm using a $432^3$ supernova data set
similar to the one used in the direct volume rendering experiments. An
example contour generated is shown in Figure~\ref{fig:Contour}. The CPU run
times are measured with two Xeon E5-2698 processors each with 16 cores and
$2\times$ hyperthreading running at 2.3~GHz. The CUDA runs are measured on
a Tesla K40. A summary of the time is given in
Table~\ref{tab:ContourTimes}. We compare the VTK-m algorithm to the similar
algorithms in PISTON and a serial algorithm in VTK that can be run in
parallel with MPI. Note that to make the comparison as authentic as
possible, we use VTK's Marching Cubes implementation of contouring
(vtkMarchingCubes) without point merging. The MPI experiments are run on a
single node.

\begin{figure}[htb]
  \centering
  \includegraphics[width=.5\linewidth]{images/Contour}
  \caption{Contour of supernova data used in isosurfacing experiments.}
  \label{fig:Contour}
\end{figure}

\begin{table}[htb]
  \centering
  \caption{Time (in seconds) to run different isosurface implementations on
    various devices with different output sizes.}
  \label{tab:ContourTimes}
  \includegraphics[width=\linewidth, trim=0.505in 0.517in 0.505in 0.505in]{data/ContourTimes}
\end{table}

In serial, our VTK-m Marching Cubes implementation runs at about half the
speed of the VTK implementation. This is likely due to duplicate
computation during the two-pass algorithm in VTK-m. However, the VTK-m
algorithm exceeds the VTK algorithm performance in parallel, almost
doubling the speed, which we attribute to there being load imbalance when
using MPI to parallelize the old VTK version. The exact same VTK-m
implementation also has good performance on the CUDA device on which it
runs faster than any of the parallel CPU implementations. The performance
of the VTK-m implementation is comparable to the previous generation
algorithm implemented in PISTON. The VTK-m implementation is slightly
slower, which we attribute to the PISTON implementation consolidating some
index computations.


\subsection{Surface Simplification}

The simplification of polygonal meshes is a critical component of
visualization applications that wish to provide rendering levels of detail.
ParaView uses a point clustering algorithm that combines mesh points using
a regular grid of bins\lcite{Lindstrom2000} as demonstrated in
Figure~\ref{fig:SurfaceSimplification}. We implement this algorithm in
VTK-m, which performs the clustering by identifying the bins in a map
operation and then combining points in the same bin by sorting based on bin
identifier.

\begin{figure}[htb]
  \centering
  \includegraphics[width=3in]{images/SurfaceSimplification}
  \caption{Example of mesh simplification. At left is the original mesh. At
    right is a mesh simplified on a $512^3$ grid of bins.}
  \label{fig:SurfaceSimplification}
\end{figure}

We demonstrate this algorithm using the ``Lucy'' data from the Stanford 3D
scanning repository (\url{http://graphics.stanford.edu/data/3Dscanrep/}).
The experiments cluster using bins of $512^3$, $1024^3$, and $2048^3$,
which reduces the output by $97\%$, $91\%$, and $67\%$, respectively. For
comparison purposes, we also demonstrate the VTK-m algorithm in serial and
the serial VTK algorithm (except for the $2048^3$ grid, for which the VTK
filter ran out of memory). The CPU run times are measured with two Intel
Xeon E5-2699 processors each with 16 cores and $2\times$ hyperthreading
running at 2.3~GHz with 64~GB of memory. The CUDA runs are measured on a
Tesla K40. A summary of the time is given in
Table~\ref{tab:SimplificationTimes}.

\begin{table}[htb]
  \centering
  \caption{Time (in seconds) to run the surface simplification algorithm on
    various devices with different output sizes. The CUDA time does not
    include data transfers to or from the device.}
  \label{tab:SimplificationTimes}
  %% \begin{tabular}{lrrrrr}
  %%   \toprule
  %%   Grid Size & TBB {\footnotesize 32 threads} & TBB {\footnotesize 64 threads} & CUDA & Serial & VTK \\
  %%   \midrule
  %%   $512^2$  & 0.36 & 0.41 & 0.18 & 2.73 & 3.65 \\
  %%   $1024^3$ & 0.45 & 0.49 & 0.19 & 2.93 & 5.22 \\
  %%   $2048^3$ & 0.72 & 0.74 & 0.20 & 5.22 & \multicolumn{1}{c}{---} \\
  %%   \bottomrule
  %% \end{tabular}
  \includegraphics[width=\linewidth, trim=0.505in 0.517in 0.505in 0.505in]{data/SimplificationTimes}
\end{table}

Even when run in serial, the VTK-m implementation of surface simplification
takes less time than the implementation in VTK. This difference is mostly
because the VTK-m algorithm is using a simpler but effective vertex
positioning that uses less computation and memory. Running the VTK-m
algorithm in parallel further reduces the running time. We do observe a
limit in the speedup for this algorithm, particularly for TBB. This appears
to be a limit of the sort algorithm provided by TBB, which dominates the
run time.

\subsection{Integration with HPC Applications}

Visualization has long been a critical tool used by scientific applications for
a variety of tasks including data exploration, validation of results, debugging,
and presentation of results. The size and complexity of the data generated by these
application codes has pushed more and more visualization onto larger, and more
massively threaded and heterogeneous resources including the supercomputer running the application
using an in situ visualization paradigm.

VTK-m is designed to assist these applications by running on both
dedicated visualization resources and
with the application on the supercomputer using an in situ paradigm.
We have been exploring the integration of VTK-m with XGC1, a highly scalable physics
code used to study plasmas in fusion tokamak devices\lcite{chang2009compressed}.
%XGC1 as a particle-in-cell code, is keenly interested in the visualization of particles
%of interest during the simulation.
Recent work has explored using light-weight plugins to perform
visualization both of the particles and of the field variables in XGC1 using
ADIOS
%\lcite{HelloADIOS}
and DataSpaces
%\lcite{docan2014}
as a lightly coupled in situ framework\lcite{BDAC:Pugmire14}.

Production runs of XGC1 produce tremendous amounts of data, particularly particle data.
XGC1 scientists are interested in visualization and analysis of these 
particles in numerous ways that include both the entire set of particles, and
various subsets of particles based on particular properties. Further, these
analyses might take place in different parts of the data workflow, from simulation
nodes, from loosely coupled nodes, or from disk. 
Depending on this workflow, the visualization and analysis code must
perform well in a variety of different locations with different computer
hardware.
VTK-m's performance portability offers this flexibility while maintaining a
low overhead.
Also, because of the size of this particle data, efficient representations on the simulation
nodes and coupling nodes is paramount.
VTK-m's data model enables an efficient representation while taking full
advantage of the data-parallel framework to process large numbers of particles.
The initial work in visualization plug-ins
was done using EAVL but is now being migrated over to the VTK-m framework.

\begin{figure}[htb]
  \centering
  \includegraphics[height=0.523\linewidth]{images/xgc_classifyPoints}
  \includegraphics[height=0.523\linewidth]{images/xgc_wallHits}
  \caption{Particle visualization from XGC1. On the left is the classification of particles
as inside (pink) or outside (white) of the plasma core. On the right the particles that interact with the
containment wall are shown in red.}
  \label{fig:XGCParticles}
\end{figure}

Figure~\ref{fig:XGCParticles} shows visualizations of particles from a recent run of XGC1.
One particular interest to the scientists is understanding the interactions between the
plasma particles and the containment vessel of the tokamak. The visualization workflow consists
of identifying particles that begin in the central core of the plasma at the start of the simulation and then
migrate over time outside the central core and collide with the tokamak wall. The visualization
plugins we are developing for this application use ADIOS for the I/O, both file based and in situ.
The in situ integration applies the VTK-m data model to the simulation particles and the tokamak wall, which allows allows VTK-m to operate on these data without copying them.
VTK-m worklets are used to first identify the particles inside and outside of the plasma
core and then to flag particles that intersect with
the tokamak wall.

We plan to continue the development of these and other visualization
plugins for XGC1 as well as
other codes running on HPC resources. We are particularly interested in furthering our
exploration of these plugins executing in loosely coupled in situ environments like ADIOS.

\section{Discussion}

Many of the scientific visualization applications in use today have roots
to software that began development over 20 years ago. Enhancements have
been added over the years to address needs of the HPC community, but these
traditional approaches do not fare well in massively threaded environments.
VTK-m fills this gap in functionality by bringing together the most recent
research.

VTK-m is not intended to be a replacement for VTK or any of the other
existing visualization tools such as ParaView and VisIt. Rather, our plan
is to integrate VTK-m with these large, existing software applications to
enhance their performance. VTK-m exists as a separate entity so that it may
be structured into the best possible framework for multithreaded
development environments.

Many visualization researchers and developers are adopting VTK-m for their
scientific visualization needs. We expect VTK-m to be a major resource for
visualization software for many years to come.

\section*{Acknowledgments}

We would like to thank Kewei Lu from The Ohio State University for valuable
contributions to the VTK-m isosurface implementation.

This material is based upon work supported by the U.S. Department of
Energy, Office of Science, Office of Advanced Scientific Computing
Research, under Award Numbers 14-017566 and 12-015215.

Sandia National Laboratories is a multi-program laboratory managed and
operated by Sandia Corporation, a wholly owned subsidiary of Lockheed
Martin Corporation, for the U.S. Department of Energy's National Nuclear
Security Administration under contract DE-AC04-94AL85000.
\hfill {\scriptsize SAND 2016-1719 J}

This manuscript has been authored by UT-Battelle, LLC under Contract
No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United
States Government retains and the publisher, by accepting the article
for publication, acknowledges that the United States Government
retains a non-exclusive, paid-up, irrevocable, world-wide license to
publish or reproduce the published form of this manuscript, or allow
others to do so, for United States Government purposes. The Department
of Energy will provide public access to these results of federally
sponsored research in accordance with the DOE Public Access Plan
(\url{http://energy.gov/downloads/doe-public-access-plan}).

\bibliographystyle{unsrt}
\bibliography{VTKmCGA2015}

\newcommand{\bio}[2]{\textit{\textbf{#1} #2}}

\bio{Kenneth Moreland}{is a principal member of technical staff at Sandia
  National Laboratories. His current research interests include large-scale
  and finely-threaded scientific visualization algorithms. Moreland
  received a PhD in computer science from the University of New Mexico.
  Contact him at kmorel@sandia.gov.}

\bio{Christopher Sewell}{is a staff scientist in the Computer and
  Computational Sciences Division at Los Alamos National Laboratory. His
  research interests include portable data-parallel programming models and
  large-scale visualization and analysis. Sewell received a PhD in computer
  science from Stanford University. Contact him at csewell@lanl.gov.}

\bio{William Usher}{is a graduate student research assistant at the
  Scientific Computing and Imaging Institute at the University of Utah. His
  current research interests include large-scale and in situ scientific
  visualization algorithms. Usher is currently pursuing a PhD in computer
  science at the University of Utah. Contact him at will@sci.utah.edu.}

\bio{Li-ta Lo}{is a technical staff member at Los Alamos National Lab. His
  research interests include task and data-parallel programming for
  scientific visualization and simulation and big data analytics. He
  received an MS in applied mechanics from the National Taiwan University.
  He can be reached by ollie@lanl.gov.}

\bio{Jeremy Meredith}{is a senior research and development staff member at
  Oak Ridge National Laboratory, focusing on high performance computing and
  large scale scientific visualization. He received his MS in computer
  science from Stanford University. Contact him at jsmeredith@ornl.gov.}

\bio{David Pugmire}{is a Senior Staff Scientist at Oak Ridge National
  Laboratory. His current research interests include methods for scientific
  visualization on high performance computers. Pugmire received a PhD in
  computer science from the University of Utah. Contact him at
  pugmire@ornl.gov.}

\bio{James Kress}{is a third year PhD Student in Computer Science at the
  University of Oregon. He received his BS in 2013 from Boise State
  University in Computer Science with a minor in Political Science. His
  current research interests include scientific visualization, high
  performance computing, and the intersection of the two. Contact him at
  jkress@cs.uoregon.edu.}

\bio{Hendrik Schroots}{is a software engineer at Intel. His research
  interests include large scale visualization and graphics. Schroots
  received his MS in visualization and graphics from the University of
  California, Davis. Contact him at hschroot@ucdavis.edu.}

\bio{Kwan-Liu Ma}{is a professor of computer science at the University of
  California, Davis. He directs VIDI Labs and UC Davis Center for
  Visualization. Professor Ma received his PhD degree in computer science
  from the University of Utah in 1993. His research interests include
  visualization, high-performance computing, and user interface design.
  Professor Ma is an IEEE Fellow and a recipient of the IEEE VGTC
  Visualization Technical Achievement Award in 2013. He presently serves as
  the AEIC of IEEE CG\&A. Contact him at ma@cs.ucdavis.edu.}

\bio{Hank Childs}{is an Associate Professor at the University of Oregon and
  a Staff Scientist at Lawrence Berkeley National Laboratory. His current
  research interests include large data visualization, visualization
  systems, and flow visualization. Childs received a PhD in computer
  science from the University of California at Davis. Contact him at
  hank@uoregon.edu.}

\bio{Matthew Larsen}{is staff scientist at Lawrence Livermore National
  Laboratory and a PhD student at the University of Oregon. His research
  interests are at the intersection of computer graphics, scientific
  visualization, and HPC. Larsen received a MS in computer science from the
  University of Oregon. Contact him at larsen30@llnl.gov.}

\bio{Chun-Ming Chen}{is a PhD candidate of the Department of Computer
  Science and Engineering at Ohio State University. His research interests
  include analysis and visualization for large flow data. He received a MS
  in Computer Science from the University of Southern California and a BS
  in Computer Science and Information Engineering from the National
  Chiao-Tung University, Taiwan. Contact him at chen.1701@osu.edu.}

\bio{Robert Maynard}{is a R\&D Engineer at Kitware Inc. He is one of the
  primary developers of VTK-m with a focus on scientific visualization on
  heterogeneous architectures. Maynard received a BS in Computer Science
  from Laurentian University. Contact him at robert.maynard@kitware.com.}

\bio{Berk Geveci}{is the Senior Director of Scientific Computing at Kitware
  Inc. His current research focuses on large scale data analysis and
  visualization within the VTK and ParaView frameworks. Geveci received a
  PhD in Mechanical Engineering from Lehigh University. Contact him at
  berk.geveci@kitware.com.}

\end{document}
