Static ArrayHandle can cause crash on exit with Kokkos
If you create a static
ArrayHandle
in a function or method, it can cause a crash on program exit with the Kokkos adapter. We have in particular seen when using the underlying HIP backend for Kokkos.
The problem is that Kokkos is finalizing itself before the ArrayHandle
. When the ArrayHandle
tries to destroy itself, it deallocates all memory on all devices. When it tries to deallocate on Kokkos/HIP, the deallocation crashes because Kokkos has already finalized HIP.
Where this gets really tricky is that Kokkos does not seem to initialize HIP during a call to initialize. Rather, the initialization gets called during the first call to an algorithm (such as parallel_for
) or creating a view in the execution environment. This means that no matter how careful you are to ensure that your ArrayHandles
get created after VTK-m is initialized (and therefore in theory cleaned up before the subsystems are done), you can still run into problems when those static ArrayHandle
s attempt to delete all their memory as they leave scope.