For all benchmarks listed above, the test bed had the clients as described in the Table 2 below. Since the number of compute nodes available for testing was only 16, when a higher number of threads was required, those threads were equally distributed on the compute nodes (i.e. 32 threads = 2 threads per node, 64 threads = 4 threads per node, 128 threads = 8 threads per node, 256 threads =16 threads per node, 512 threads = 32 threads per node, 1024 threads = 64 threads per node). The intention was to simulate a higher number of concurrent clients with the limited number of compute nodes available. Since some benchmarks support a high number of threads, a maximum value up to 1024 was used (specified for each test), while avoiding excessive context switching and other related side effects from affecting performance results.

Table 2 Client test bed

Sequential IOzone Performance N clients to N files

Sequential N clients to N files performance was measured with IOzone version 3.487. Tests executed varied from single thread up to 1024 threads in increments of powers of two.

Caching effects on the servers were minimized by setting the GPFS page pool tunable to 16GiB and using files bigger than two times that size. It is important to notice that for GPFS that tunable sets the maximum amount of memory used for caching data, regardless the amount of RAM installed and free. Also, important to notice is that while in previous DellEMC HPC solutions the block size for large sequential transfers is 1 MiB, GPFS was formatted with 8 MiB blocks and therefore that value is used on the benchmark for optimal performance. That may look too large and apparently waste too much space, but GPFS uses subblock allocation to prevent that situation. In the current configuration, each block was subdivided in 256 subblocks of 32 KiB each.

The following commands were used to execute the benchmark for writes and reads, where $Threads was the variable with the number of threads used (1 to 1024 incremented in powers of two), and threadlist was the file that allocated each thread on a different node, using round robin to spread them homogeneously across the 16 compute nodes.

To avoid any possible data caching effects from the clients, the total data size of the files was twice the total amount of RAM in the clients used. That is, since each client has 128 GiB of RAM, for threads counts equal or above 16 threads the file size was 4096 GiB divided by the number of threads (the variable $Size below was used to manage that value). For those cases with less than 16 threads (which implies each thread was running on a different client), the file size was fixed at twice the amount of memory per client, or 256 GiB.

iozone -i0 -c -e -w -r 8M -s $ G -t $Threads -+n -+m ./threadlist
iozone -i1 -c -e -w -r 8M -s $ G -t $Threads -+n -+m ./threadlist

Figure 2  N to N Sequential Performance

From the results we can observe that write performance rises with the number of threads used and then reaches a plateau at around 64 threads for writes and 128 threads for reads. Then read performance also rises fast with the number of threads, and then stays stable until the maximum number of threads that IOzone allow is reached, and therefore large file sequential performance is stable even for 1024 concurrent clients. Write performance drops about 10% at 1024 threads. However, since the client cluster has less than that number of cores, it is uncertain if the drop in performance is due to swapping and similar overhead not observed in spinning media(since NVMe latency is very low compared to spinning media), or if the RAID 10 data synchronization is becoming a bottleneck. More clients are needed to clarify that point. An anomaly on the reads was observed at 64 threads, where performance did not scale at the rate that was observed for previous data points, and then on the next data point moves to a value very close to the sustained performance. More testing is needed to find the reason for such anomaly but is out of the scope of this blog.

The maximum read performance for reads was below the theoretical performance of the NVMe devices (~102 GB/s), or the performance of EDR links, even assuming that one link was mostly used for NVMe over fabric traffic (4x EDR BW ~96 GB/s).
However, this is not a surprise since the hardware configuration is not balanced with respect to the NVMe devices and IB HCAs under each CPU socket. One CX6 adapter is under CPU1, while the CPU2 has all the NVMe devices and the second CX6 adapters. Any storage traffic using the first HCA must use the UPIs to access the NVMe devices. In addition, any core in CPU1 used must access devices or memory assigned to CPU2, so data locality suffers, and UPI links are used. That can explain the reduction for the maximum performance, compared to the max performance of the NVMe devices or the line speed for CX6 HCAs. The alternative to fix that limitation is having a balanced hardware configuration which implies reducing density to half by using an R740 with four x16 slots and use two x16 PCIe expanders to equally distribute NVMe devices on two CPUs and having one CX6 HCA under each CPU.

Sequential IOR Performance N clients to 1 file

Sequential N clients to a single shared file performance was measured with IOR version 3.3.0, assisted by OpenMPI v4.0.1 to run the benchmark over the 16 compute nodes. Tests executed varied from a one thread up to 512 threads since there were not enough cores for 1024 or more threads. This benchmark tests used 8 MiB blocks for optimal performance. The previous performance test section has a more complete explanation for why that matters.

Data caching effects were minimized by setting the GPFS page pool tunable to 16GiB and the total file size was twice the total amount of RAM in the clients used. That is, since each client has 128 GiB of RAM, for threads counts equal or above 16 threads the file size was 4096 GiB, and an equal amount of that total was divided by the number of threads (the variable $Size below was used to manage that value). For those cases with less than 16 threads (which implies each thread was running on a different client), the file size was twice the amount of memory per client used times the number of threads, or in other words, each thread was asked to use 256 GiB.

The following commands were used to execute the benchmark for writes and reads, where $Threads was the variable with the number of threads used (1 to 1024 incremented in powers of two), and my_hosts.$Threads is the corresponding file that allocated each thread on a different node, using round robin to spread them homogeneously across the 16 compute nodes.

mpirun --allow-run-as-root -np $Threads --hostfile my_hosts.$Threads --mca btl_openib_allow_ib 1 --mca pml ^ucx --oversubscribe --prefix /mmfs1/perftest/ompi /mmfs1/perftest/lanl_ior/bin/ior -a POSIX -v -i 1 -d 3 -e -k -o /mmfs1/perftest/tst.file -w -s 1 -t 8m -b $ G

mpirun --allow-run-as-root -np $Threads --hostfile my_hosts.$Threads --mca btl_openib_allow_ib 1 --mca pml ^ucx --oversubscribe --prefix /mmfs1/perftest/ompi /mmfs1/perftest/lanl_ior/bin/ior -a POSIX -v -i 1 -d 3 -e -k -o /mmfs1/perftest/tst.file -r -s 1 -t 8m -b $ G

Figure 3  N to 1 Sequential Performance

From the results we can observe read and write performance are high regardless of the implicit need for locking mechanisms since all threads access the same file. Performance rises again very fast with the number of threads used and then reaches a plateau that is relatively stable for reads and writes all the way to the maximum number of threads used on this test. Notice that the maximum read performance was 51.6 GB/s at 512 threads, but the plateau in performance is reach at about 64 threads. Similarly, notice that the maximum write performance of 34.5 GB/s was achieved at 16 threads and reached a plateau that can be observed until the maximum number of threads used.

Random small blocks IOzone Performance N clients to N files

Random N clients to N files performance was measured with IOzone version 3.487. Tests executed varied from single thread up to 1024 threads in increments of powers of two.

Tests executed varied from single thread up to 512 threads since there was not enough client-cores for 1024 threads. Each thread was using a different file and the threads were assigned round robin on the client nodes. This benchmark tests used 4 KiB blocks for emulating small blocks traffic and using a queue depth of 16. Results from the large size solution and the capacity expansion are compared.

Caching effects were again minimized by setting the GPFS page pool tunable to 16GiB and to avoid any possible data caching effects from the clients, the total data size of the files was twice the total amount of RAM in the clients used. That is, since each client has 128 GiB of RAM, for threads counts equal or above 16 threads the file size was 4096 GiB divided by the number of threads (the variable $Size below was used to manage that value). For those cases with less than 16 threads (which implies each thread was running on a different client), the file size was fixed at twice the amount of memory per client, or 256 GiB.

iozone -i0 -I -c -e -w -r 8M -s $ G -t $Threads -+n -+m ./nvme_threadlist                                     <= Create the files sequentially
iozone -i2 -I -c -O -w -r 4k -s $ G -t $Threads -+n -+m ./nvme_threadlist                                      <= Perform the random reads and writes.

Figure 4  N to N Random Performance

From the results we can observe that write performance starts at a high value of 6K IOps and rises steadily up to 1024 threads where it seems reach a plateau with over 5M IOPS if more threads could be used. Read performance on the other hand starts at 5K IOPSs and increases performance steadily with the number of threads used (keep in mind that number of threads is doubled for each data point) and reaches the maximum performance of 7.3M IOPS at 1024 threads without signs of reaching a plateau. Using more threads will require more than the 16 compute nodes to avoid resources starvation and excessive swapping that can lower apparent performance, where the NVMe nodes could in fact maintain the performance.

Metadata performance with MDtest using 4 KiB files

Metadata performance was measured with MDtest version 3.3.0, assisted by OpenMPI v4.0.1 to run the benchmark over the 16 compute nodes. Tests executed varied from single thread up to 512 threads. The benchmark was used for files only (no directories metadata), getting the number of creates, stats, reads and removes the solution can handle, and results were contrasted with the Large size solution.

The optional High Demand Metadata Module was used, but with a single ME4024 array, even that the large configuration and tested in this work was designated to have two ME4024s. The reason for using that metadata module is that currently these NVMe nodes are used as Storage targets for data only. However, the nodes could be used to store data and metadata, or even as a flash alternative for the high demand metadata module, if extreme metadata demands call for it. Those configurations were not tested as part of this work.

Since the same High Demand Metadata module has been used for previous benchmarking of the DellEMC Ready Solution for HPC PixStor Storage solution, metadata results will be very similar compared to previous blog results. For that reason, the study with empty files was not done, and instead 4 KiB files were used. Since 4KiB files cannot fit into an inode along with the metadata information, NVMe nodes will be used to store data for each file. Therefore, MDtest can give a rough idea of small files performance for reads and the rest of the metadata operations.

The following command was used to execute the benchmark, where $Threads was the variable with the number of threads used (1 to 512 incremented in powers of two), and my_hosts.$Threads is the corresponding file that allocated each thread on a different node, using round robin to spread them homogeneously across the 16 compute nodes. Similar to the Random IO benchmark, the maximum number of threads was limited to 512, since there are not enough cores for 1024 threads and context switching would affect the results, reporting a number lower than the real performance of the solution.

mpirun --allow-run-as-root -np $Threads --hostfile my_hosts.$Threads --prefix /mmfs1/perftest/ompi --mca btl_openib_allow_ib 1 /mmfs1/perftest/lanl_ior/bin/mdtest -v -d /mmfs1/perftest/ -i 1 -b $Directories -z 1 -L -I 1024 -y -u -t -F -w 4K -e 4K

Since performance results can be affected by the total number of IOPS, the number of files per directory and the number of threads, it was decided to keep fixed the total number of files to 2 MiB files (2^21 = 2097152), the number of files per directory fixed at 1024, and the number of directories varied as the number of threads changed as shown in Table 3.

Table 3 MDtest distribution of files on directories

Figure 5  Metadata Performance – 4 KiB Files

First, notice that the scale chosen was logarithmic with base 10, to allow comparing operations that have differences with several orders of magnitude; otherwise some of the operations would look like a flat line close to 0 on a  linear scale. A log graph with base 2 could be more appropriate, since the number of threads are increased in powers of 2, but the graph would look very similar and people tend to handle and remember better numbers based on powers of 10.

The system gets very good results as previously reported with Stat operations reaching the peak value at 64 threads with almost 6.9M op/s and then is reduced for higher thread counts reaching a plateau. Create operations reach the maximum of 113K op/s at 512 threads, so is expected to continue increasing if more client nodes (and cores) are used. Reads and Removes operations attained their maximum at 128 threads, achieving their peak at almost 705K op/s for Reads and 370K op/s for removes, and then they reach plateaus. Stat operations have more variability, but once they reach their peak value, performance does not drop below 3.2M op/s for Stats. Create and Removal are more stable once their reach a plateau and remain above 265K op/s for Removal and 113K op/s for Create. Finally, reads reach a plateau with performance above 265K op/s.

Conclusions and Future Work

The NVMe nodes are an important addition to the HPC storage solution with to provide a very high-performance tier, with good density, very high random-access performance, and very high sequential performance. Furthermore, the solution scales out in capacity and performance linearly as more NVMe nodes modules are added. The performance from the NVMe nodes can be overviewed on Table 4, is expected to be stable and those values can be used to estimate performance for a different number of NVMe nodes.
However, keep in mind that each pair of NVMe nodes will provide half of any number shown in Table 4.
This solution provides HPC customers with a very reliable parallel file system used by many Top 500 HPC clusters. In addition, it provides exceptional search capabilities, advanced monitoring and management, and adding optional gateways allow file sharing via ubiquitous standard protocols like NFS, SMB and others to as many clients as needed.

Table 4  Peak & Sustained Performance for 2 Pairs of NVMe nodes

Since NVMe nodes were only used for data, possible future work can include using them for data and metadata and have a self-contained flash-based tier with better metadata performance due to the higher bandwidth and lower latency of NVMe devices compared to SAS3 SSDs behind RAID controllers. Alternatively, if a customer has extremely high metadata demands and require a solution more dense than what the high demand metadata module can provide, some or all of the distributed RAID 10 devices can be used for metadata in the same way that RAID 1 devices on the ME4024s are used now.
Another blog to be release soon will characterize the PixStor Gateway nodes, which allow connecting the PixStor solution to other networks using NFS or SMB protocols and can scale out performance. Also, the solution will be updated to HDR100 very soon, and another blog is expected to talk about that work.