VTKM CUDA code hangs sometimes on Summit; CPU version completes
On Summit, using VTK-m, CUDA Version 10.1.168, we have ran the following code, in particular, we ran isosurface_nompi.cpp
in the linked repo.
The CPU version completes without issue. However, the GPU version hangs sometimes. The following is the output from cuda-memcheck
========= Program hit cudaErrorInvalidValue (error 1) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib64/libcuda.so.1 [0x461d2c]
========= Host Frame:./isosurface_NOMPI_VTKM [0x2867c]
========= Host Frame:/usr/lib64/libc.so.6 [0x25200]
========= Host Frame:/usr/lib64/libc.so.6 (__libc_start_main + 0xc4) [0x253f4]
=========
========= Program hit cudaErrorInvalidValue (error 1) due to "invalid argument" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib64/libcuda.so.1 [0x461d2c]
========= Host Frame:./isosurface_NOMPI_VTKM [0xe5fe9c]
...
========= Host Frame:./isosurface_NOMPI_VTKM [0x2867c]
========= Host Frame:/usr/lib64/libc.so.6 [0x25200]
========= Host Frame:/usr/lib64/libc.so.6 (__libc_start_main + 0xc4) [0x253f4]
=========
Please suggest what could be the possible cause of the hanging and how can this be resolved.