CentOS 8 on DELL C4140/4x Nvidia Tesla V100 GPU fails

Hello,

I am running CentOS 8 on a DELL C4140 with 4 NVIDIA Tesla V100 GPUs:

1. # nvidia-smi -L
2. GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-ca51cbd1-4eb0-f265-9cb6-613f848d9ebd)
3. GPU 1: Tesla V100-SXM2-32GB (UUID: GPU-d02431f6-3ad3-e257-2858-8e830e06fa56)
4. GPU 2: Tesla V100-SXM2-32GB (UUID: GPU-2c34530c-3db6-3cd2-df6b-39f65d3448c2)
5. GPU 3: Tesla V100-SXM2-32GB (UUID: GPU-c8740a89-14af-b1f3-d17c-629cb2454737)

The installed NVIDIA driver version 440.33 seems to load fine in persistance mode only. nvidia-smi reports the four GPUs as follows:

 

1. # nvidia-smi
2. Tue Dec 17 15:07:11 2019
3. +-----------------------------------------------------------------------------+
4. | NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
5. |-------------------------------+----------------------+----------------------+
6. | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
7. | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
8. |===============================+======================+======================|
9. | 0 Tesla V100-SXM2... Off | 00000000:1A:00.0 Off | 0 |
10. | N/A 36C P0 57W / 300W | 0MiB / 32510MiB | 0% Default |
11. +-------------------------------+----------------------+----------------------+
12. | 1 Tesla V100-SXM2... Off | 00000000:1C:00.0 Off | 0 |
13. | N/A 34C P0 57W / 300W | 0MiB / 32510MiB | 0% Default |
14. +-------------------------------+----------------------+----------------------+
15. | 2 Tesla V100-SXM2... Off | 00000000:1D:00.0 Off | 0 |
16. | N/A 32C P0 54W / 300W | 0MiB / 32510MiB | 0% Default |
17. +-------------------------------+----------------------+----------------------+
18. | 3 Tesla V100-SXM2... Off | 00000000:1E:00.0 Off | 0 |
19. | N/A 34C P0 58W / 300W | 0MiB / 32510MiB | 3% Default |
20. +-------------------------------+----------------------+----------------------+
21.  
22. +-----------------------------------------------------------------------------+
23. | Processes: GPU Memory |
24. | GPU PID Type Process name Usage |
25. |=============================================================================|
26. | No running processes found |
27. +-----------------------------------------------------------------------------+

 

The error logs contain the following message:

1. NVRM: GPU at PCI:0000:1e:00: GPU-c8740a89-14af-b1f3-d17c-629cb2454737
2. NVRM: GPU Board Serial Number: 0324518173584
3. NVRM: Xid (PCI:0000:1e:00 62, pid=10230, 0a76(2d50) 00000000 00000000

According to the list of Xid errors (see https://docs.nvidia.com/deploy/xid-errors/index.html), error 62 refers to an internal micro-controller halt (newer drivers), and is related to hardware error, driver error or thermal issue.

How do I identify the GPU that is not working properly? Taking off the cooler of the GPU does not show the GPU id, and removing the GPU from the board in order to have a look under it voids the warranty of the server.

Any help on that is appreciated.

Thanking in advance,

Frank