Dell PowerScale and NVIDIA GPUDirect Performance Report
Executive summary
Overview
Revisions
We value your feedback
Introduction
Introduction
Solution architecture
Architecture overview
Figure 1 illustrates the architecture, showing the key components that made up the solution as it was tested and benchmarked.
Figure 1. Architecture overview
Storage: Dell PowerScale scale-out NAS
Networking: Dell PowerSwitch data center switches
Bill of materials
Benchmark
Benchmark methodology
Benchmark results
Figure 2 shows the results of the GDSIO tool with the following parameters:
- Sequential READs
- 512 KiB IO Size
- 8 threads per GPU
- 256 GB file size
Figure 2 also includes results numbers from previous GDSIO tests using OneFS 9.2 and NVIDIA V100 GPUs.
Figure 2. GDSIO performance results comparison between OneFS 9.2 and OneFS 9.5
F600 performance nodes with upgraded Intel Gold 6248R processors, memory, and OneFS 9.5 showed a 36 percent performance improvement over previous GDSIO testing with standard F600 nodes. This significant increase in performance is the result of higher performant components within the F600, such as upgraded Intel processors, as well as performance improvements embedded into OneFS 9.5.
This increase in node performance increases the overall performance density of a PowerScale cluster. A single F600 node is one rack unit or 1RU, and a minimum cluster size is three nodes or 3RU. This equates to over 24 GB/s per minimum cluster throughput, scaling linearly up to 252 nodes in a cluster. A fully populated F600 PowerScale cluster can serve over 2 TB/s of read throughput to a massive GPU farm. For companies looking to build supercomputers or extremely large generative AI models, PowerScale is an excellent platform choice.
Figure 3. Performance benefits with GDSIO and F600 performance nodes
Understanding NUMA node affinity and PCIe tree
PCIe topology
The PCIe topology, PCIe root complex, and the physical location of the GPU and network and storage devices are the most important things to understand with GPUDirect. The key is to limit the number of “hops” for a GPU to communicate with a NIC (and group GPUs and NICs based on their NUMA node affinity), also called CPU affinity. To retrieve this information, you could use some Linux commands like lspci or lstopo to identify PCIe devices, referred to as BDF notation (bus:device.func). However, the NVIDIA CUDA Toolkit package provides a command called nvidia-smi that can facilitate this task. See Figure 4 for details.
Figure 4. PCIe topology
Based on nvidia-smi topo -mp command output (Figure 4), we can determine the following mapping for the R7525s used in this testing.
Table 3. PCIe mapping
|
GPU ID |
Mellanox card |
Numa affinity |
|
0 |
mlx5_0 |
1 |
|
1 |
mlx5_0 |
1 |
Note: You can use the ibdev2netdev command to see which interface names (mlx5_X) map to which device names (enpYsZ).
Example:
# ibdev2netdev
mlx5_0 port 1 ==> ens6f0 (Up)
mlx5_1 port 1 ==> ens6f1 (Up)
Configuration details
PowerScale configuration
Networking configuration
There are two subnets, each with one pool. One subnet is for 100gige-1 (ext-a) and the other is for 100gige-2 (ext-b). In total, fourteen (14) 100 GbE ports are used in the PowerScale cluster.
All 100gige-1 interfaces are connected to the first switch, and all 100gige-2 interfaces are connected to the second switch.
F600GDSBench-1# isi network interfaces list
LNN Name Status VLAN ID Owners Owner Type IP Addresses
-----------------------------------------------------------------------------------
1 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.101
1 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.101
2 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.102
2 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.102
3 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.103
3 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.103
4 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.104
4 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.104
5 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.105
5 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.105
6 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.106
6 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.106
7 100gige-1 Up - groupnet0.subnet10.pool10 Static 10.100.10.107
7 100gige-2 Up - groupnet0.subnet20.pool20 Static 10.100.20.107
-----------------------------------------------------------------------------------
Total: 14
Disable compression
During tests, compression was disabled.
isi compression settings modify --enabled=0
Disable inline deduplication
During tests, inline deduplication was disabled.
isi dedupe inline settings modify --mode=disabled
Disable endurant cache (EC)
During tests, Endurant Cache was disabled.
isi_for_array sysctl efs.bam.ec.mode=0
Set file pool policy to streaming
F600GDSBench-1# cd /ifs/benchmark
F600GDSBench-1# ls
gdsio
F600GDSBench-1# isi set -l streaming gdsio
F600GDSBench-1# isi set -a streaming gdsio
F600GDSBench-1# isi get -d gdsio
POLICY LEVEL PERFORMANCE COAL FILE
default 4x streaming/@18 on gdsio/
Enable jumbo frames on each subnet
Figure 5. Subnet configuration (10.100.10.0/24)
Figure 6. Subnet configuration (10.100.20.0/24)
Enable NFS Over RDMA on each network pool
Figure 7. Pool configuration for subnet 10.100.10.0/24
Figure 8. Pool configuration for subnet 10.100.20.0/24
Enable NFS Over RDMA on NFS global settings
Figure 9. NFS global settings