CCE AI Suite (NVIDIA GPU)

Introduction

Notes and Constraints

• The driver to be downloaded must be a .run file.
• Only NVIDIA Tesla drivers are supported, not GRID drivers.
• When installing or reinstalling the add-on, ensure that the driver download address is correct and accessible. CCE does not verify the address validity.
• The gpu-beta add-on only enables you to download the driver and execute the installation script. The add-on status only indicates that how the add-on is running, not whether the driver is successfully installed.
• CCE does not guarantee the compatibility between the GPU driver version and the CUDA library version of your application. You need to check the compatibility by yourself.
• If a custom OS image has had a GPU driver installed, CCE cannot ensure that the GPU driver is compatible with other GPU components such as the monitoring components used in CCE.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons, locate CCE AI Suite (NVIDIA GPU) on the right, and click Install.
2. Determine whether to enable Use DCGM-Exporter to Observe DCGM Metrics. After this function is enabled, DCGM-Exporter is deployed on the GPU node.
   Notice

   If the add-on version is 2.7.40 or later, DCGM-Exporter can be deployed. DCGM-Exporter maintains the community capability and does not support the sharing mode or GPU virtualization.

3. Configure the add-on parameters.

   Table 1 Add-on parameters

   Parameter

   Description

   Default Cluster Driver

   All GPU nodes in a cluster use the same driver. You can select a proper GPU driver version or customize the driver link and enter the download link of the NVIDIA driver. NOTICE: If the download link is a public network address, for example, https://us.download.nvidia.com/tesla/470.103.01/NVIDIA-Linux-x86_64-470.103.01.run, bind an EIP to each GPU node. For details about how to obtain the driver link, see Obtaining the Driver Link from Public Network. If the download link is an OBS URL, you do not need to bind an EIP to GPU nodes. For details about how to obtain the driver link, see Obtaining the Driver Link from OBS. Ensure that the NVIDIA driver version matches the GPU node. If the driver version is changed, restart the node to apply the change.

   Note

   After the add-on is installed, you can configure GPU virtualization and node pool drivers on the Heterogeneous Resources tab in Settings.

4. Click Install.
   Note

   If the add-on is uninstalled, GPU pods newly scheduled to the nodes cannot run properly, but GPU pods already running on the nodes will not be affected.

Verifying the Add-on

• GPU node:

  # If the add-on version is earlier than 2.0.0, run the following command:
  cd /opt/cloud/cce/nvidia/bin && ./nvidia-smi
  
  # If the add-on version is 2.0.0 or later and the driver installation path is changed, run the following command:
  cd /usr/local/nvidia/bin && ./nvidia-smi
  
  

• Container:

  cd /usr/local/nvidia/bin && ./nvidia-smi
  
  

Obtaining the Driver Link from Public Network

1. Log in to the CCE console.
2. Create a node. In the Specifications area, select the GPU node flavor. The GPU card models are displayed in the lower part of the area.

3. Log in to the NVIDIA driver download page and search for the driver information. The OS must be Linux 64-bit.

   Figure 1 Selecting parameters

   
   

4. After confirming the driver information, click Find. On the displayed page, find the driver to be downloaded and click View.

   Figure 2 Viewing the driver information

   
   

5. Click Download and copy the download link.

   Figure 3 Obtaining the link

   
   

Obtaining the Driver Link from OBS

1. Upload the driver to OBS and set the driver file to public read.
   Note

   When the node is restarted, the driver will be downloaded and installed again. Ensure that the OBS bucket link of the driver is valid.

2. In the bucket list, click a bucket name, and then the Overview page of the bucket is displayed.
3. In the navigation pane, choose Objects.
4. Select the name of the target object and copy the driver link on the object details page.

Components

GPU Metrics