Skip to content

Device Resource Management

The current device management in VTK-m only allows the client program to select a type of device. However, production applications need more fine-grained control. For example, some ECP platforms have multiple GPU cards in a single node, and different client programs might need different ways of managing what gets run on each. Currently, the only way to get information about and set parameters for a device is to circumvent VTK-m and modify parameters directly. Instead, VTK-m should provide a generic way to control parameters for devices.

Parameters Offered

The following parameters to configure the behavior on different devices. These parameters borrow heavily from the Kokkos project as they have thought about this problem already. (We will also rely on their implementation for ourselves.)

It should be noted that each device will ignore parameters that do not apply to them (or that are otherwise not supported).

  • Threads Specify the number of threads to execute on. If NUMA regions are not specified, this is the total number of threads. If NUMA regions are specified, then this is the number of threads per NUMA region. (Might be more clear if it is always just the total and then divide them among NUMA regions.)
  • NUMA Specify the number of NUMA regions to use by the process. The execution will span these NUMA regions. To partition among NUMA regions, you will need to use multiple processes.
  • Device Instance The index for the instance of the device to use. For example, if you have 4 GPUs (of the same type) on a node, you can use this parameter to select which of these GPUs to use.

Another helpful feature provided by Kokkos is the ability to specify the number of devices and have the system automatically assign devices based on round-robin assignment from the MPI rank. This sounds like a nice feature, but I'm not sure how to implement it for any device other than Kokkos. We might address that in a later version. The trick is to get a unique rank for processes on the same node.

Mechanisms for Setting the Configuration

There will be multiple ways in which to set options for the configuration of a device.

Runtime Device Configuration Class

You will be able to set the configuration parameters through a class named RuntimeDeviceConfiguration. This class will have (virtual) methods with names like SetThreads and SetDeviceInstance that allow you to set the configuration parameters previously mentioned.

Each device type has its own instance of RuntimeDeviceConfiguration. This specific class cannot be created or copied. An instance of RuntimeDeviceConfiguration will be retrieved using RuntimeDeviceInformation::GetConfiguration(). This method takes as an argument the device to get the configuration for.

Internally, RuntimeDeviceInformation creates the RuntimeDeviceConfiguration in a way similar to DeviceAdaperMemoryManager. Each device adapter will create a specialization of the RuntimeDeviceConfigurationImpl, which must be a subclass of RuntimeDeviceConfiguration. RuntimeDeviceInformation then, for each device in VTKM_DEFAULT_DEVICE_ADAPTER_LIST creates an instance of RuntimeDeviceConfigurationImpl for that device tag.

Although applications are free to use this class themselves, it is more likely to set the configuration through other the mechanisms. The main function of this simple configuration class is so that device implementations can provide a simple way to configure themselves without having to parse out all of the configuration options.

That said, some devices, such as Kokkos, have their own special command line arguments that might be important or otherwise need to be initialized. So additionally RuntimeDeviceConfiguration will have an Initialize method that can be called by vtkm::cont::Initialize.

Command Line Arguments

VTK-m already has a convenient method of specifying configuration through command line arguments, so we will expand that to include arguments for resource management. The arguments will have names like --vtkm-num-threads and --vtkm-numa-regions.

As part of this change, we will move to prefix all VTK-m command line arguments with --vtkm- to differentiate them from possible application command line arguments. This will allow an application to call vtkm::cont::Initialize, which will remove any VTK-m-specific argument without worrying about removing an argument the application expects. It also makes it easier for applications to detect VTK-m arguments and ignore them. (See #597 (closed).)

To make sure that each device is ready to be configured and to make sure that any device-specific command line arguments are already processed, vtkm::cont::Initialize will first call initialize on the RuntimeDeviceConfiguration for all valid devices. It will then parse the VTK-m arguments and call methods in the RuntimeDeviceConfigurations as appropriate.

Environment Variables

Environment variables are an important mechanism to set configuration parameters. They can be used when the VTK-m library is buried under other software layers. It is also extremely helpful in HPC environments where configuration is often set up through environment variables.

Each configuration parameter will have associated with it an expected environment variable such as VTKM_NUM_THREADS or VTKM_NUMA_REGIONS. When vtkm::cont::Initialize is called, it will look for each of these environment variables and set the configuration accordingly. The specific order of operations will be for vtkm::cont::Initialize to initialize the RuntimeDeviceConfigurations, then to set any parameters established through environment variables, and then to parse the command line arguments.

(Note that with this setup, the environment variables are only examined when vtkm::cont::Initialize is called. Strictly speaking, calling vtkm::cont::Initialize is optional, but if it is not called then the environment variables will not be respected.)

Working with Kokkos Configuration

Kokkos configures parameters for its devices only through command line options and environment variables. Furthermore, it only checks for this configuration on the first call to Kokkos::initialize.

This creates a chicken-and-egg problem where we expect to call Kokkos::initialize before checking environment variables and command line arguments, but then there is no way to set these options in Kokkos.

To get around this problem, RuntimeDeviceConfiguration::Initialize for Kokkos will not immediately call Kokkos::initialize. Instead, it will search the command line arguments for strings starting with --kokkos. It will remove these arguments from the passed argv and stash them.

When one of the other methods in RuntimeDeviceConfiguration is used to set a configuration option for Kokkos, it first check to see if Kokkos has already been configured. If it has, then a warning is logged. Otherwise, it sets the environment variable associated with the configuration option.

When vtkm::cont::Initialize returns, Kokkos is still not technically initialized (unless initialized by something else). This gives the calling code more chances to set configuration parameters.

Whenever a vtkm::cont::DeviceAdapterAlgorithm method is called for the Kokkos device, it first checks to see if Kokkos has been initialized (using Kokkos::is_initialized(). If it has not been, then it calls Kokkos::initialize using any stashed arguments.