Skip to content

CUDA/NVHPC: nvhpc 22.9 multi-cuda selects old SDK for new driver

The job added by !7946 (merged) started failing CUDA tests in the CI job when the CI hosts were updated from NVIDIA driver version 520 to version 525.

Somehow the cuda_multi dispatching nvcc tool, with driver version 525, selects CUDA 10.2:

$ nvidia-smi | grep Version
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 10.2, V10.2.89

With driver versions 520 and earlier, the same tool selects CUDA 11.7:

$ nvidia-smi | grep Version
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 11.7, V11.7.99
$ nvidia-smi | grep Version
| NVIDIA-SMI 510.85.02    Driver Version: 510.85.02    CUDA Version: 11.6     |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 11.7, V11.7.99
Edited by Brad King
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information