Story #17240

Updated by Peter Amstutz 2 months ago

https://docs.nvidia.com/deploy/cuda-compatibility/index.html

h1. Nvidia says

The CUDA software environment consists of three parts:

* CUDA Toolkit (libraries, CUDA runtime and developer tools) - User-mode SDK used to build CUDA applications
* CUDA driver - User-mode driver component used to run CUDA applications (such as libcuda.so on Linux systems)
* NVIDIA GPU device driver - Kernel-mode driver component for NVIDIA GPUs

On Linux systems, the CUDA driver and kernel mode components are delivered together in the NVIDIA display driver package. This is shown in Figure 1.

...

1.3. Binary Compatibility

We define binary compatibility as a set of guarantees provided by the library, where an application targeting the said library will continue to work when dynamically linked against a different version of the library.

The CUDA Driver API has a versioned C-style ABI, which guarantees that applications that were running against an older driver (for example CUDA 3.2) will still run and function correctly against a modern driver (for example one shipped with CUDA 11.0). This is a stronger contract than an API guarantee - an application might need to change its source when recompiling against a newer SDK, but replacing the driver with a newer version will always work.

The CUDA Driver API thus is binary-compatible (the OS loader can pick up a newer version and the application continues to work) but not source-compatible (rebuilding your application against a newer SDK might require source changes). In addition, the binary-compatibility is in one direction: backwards.

...

Each version of the CUDA Toolkit (and runtime) requires a minimum version of the NVIDIA driver. The CUDA driver (libcuda.so on Linux for example) included in the NVIDIA driver package, provides binary backward compatibility. For example, an application built against the CUDA 3.2 SDK will continue to function even on today’s driver stack. On the other hand, the CUDA runtime has not provided either source or binary compatibility guarantees. Newer major and minor versions of the CUDA runtime have frequently changed the exported symbols, including their version or even their availability, and the dynamic form of the library has its shared object name (.SONAME in Linux-based systems) change every minor version.

h1. Notes

Inside the container: must include the correct nvidia runtime (if dynamically linked) or the application must be statically linked.

* runtime requires a minimum version of the driver -- this should be declared as a requirement
* nvidia-smi tells us some stuff?
* cubins (programs compiled directly for a GPU) target a specific "compute" capability and are only backwards compatible across minor revisions.
* required libraries: libcuda.so.* - the CUDA Driver
* required libraries: libnvidia-ptxjitcompiler.so.*

Apparently nvidia also offers "driver containers" where it actually installs the kernel driver (???) on the fly and a persistent daemon, instead of relying on drivers being installed on the host.

https://docs.nvidia.com/datacenter/cloud-native/driver-containers/overview.html

* to use docker, need to install https://github.com/NVIDIA/nvidia-docker
** https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html
** I still haven't figured out the correct way to invoke gpu support via the Docker API that we use in crunch-run

* Singularity support for bind mounting the nvidia driver exists. It apparently requires @nvidia-container-cli@
https://github.com/NVIDIA/libnvidia-container
Seems like you can use this tool to interrogate the system to find out which libraries need to be bind mounted.

node needs to declare, for each device:
* driver version
* hardware capability

container request needs to specify
* minimum driver version → select nodes with >= minimum driver version
* cubin hardware capability (or none) → select nodes with = major revision, >= minor revision hardware capability
** can compile for multiple targets, so this should be a list
* PTX hardware capability → select nodes with >= PTX hardware capability

It seems each SDK release has a new driver, so often the SDK version is printed instead of the underlying driver version. There's a table of which driver corresponds to which SDK revision.

With version 11 of the SDK it seems that the userspace part of the driver can be upgraded without upgrading the kernel driver.

h2. What docker --gpus does

https://github.com/docker/cli/blob/88c6089300a82d3373892adf6845a4fed1a4ba8d/docs/reference/commandline/run.md

https://docs.docker.com/config/containers/resource_constraints/

https://github.com/docker/cli/blob/88c6089300a82d3373892adf6845a4fed1a4ba8d/opts/gpus.go

Using the API, the simplest valid request is @DeviceRequest@ with:
* DeviceRequest.Count = 1
* DeviceRequest.Capabilities = [["gpu"]]

A better one is probably:
* DeviceRequest.Driver = "nvidia"
* DeviceRequest.Count = -1 (request all GPUs)
* DeviceRequest.Capabilities = [["gpu", "nvidia", "compute"]]

Docker sets NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES

Driver capabilities and options:

https://github.com/nvidia/nvidia-container-runtime#supported-driver-capabilities

Back