CUDA/NVHPC: nvhpc 22.9 multi-cuda selects old SDK for new driver
The job added by !7946 (merged) started failing CUDA tests in the CI job when the CI hosts were updated from NVIDIA driver version 520 to version 525.
Somehow the cuda_multi
dispatching nvcc
tool, with driver version 525, selects CUDA 10.2:
$ nvidia-smi | grep Version
| NVIDIA-SMI 525.60.11 Driver Version: 525.60.11 CUDA Version: 12.0 |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 10.2, V10.2.89
With driver versions 520 and earlier, the same tool selects CUDA 11.7:
$ nvidia-smi | grep Version
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 11.7, V11.7.99
$ nvidia-smi | grep Version
| NVIDIA-SMI 510.85.02 Driver Version: 510.85.02 CUDA Version: 11.6 |
$ docker run -it --gpus=all nvcr.io/nvidia/nvhpc:22.9-devel-cuda_multi-ubuntu22.04 nvcc --version
Cuda compilation tools, release 11.7, V11.7.99
Edited by Brad King