R630 - artificial disk write speed limits on SATA

Question

We notice that SATA speeds on our R630 have sequential writes capped depending on performance profile used, plus additional constant base 100MB/s reduction.
Specs are 2 x E5-2697v3 CPU / 128GB RAM / 8 x 2.5" BP and 3 SSD disks from different manufacturers:

- Micron M600 SSD 256GB
- Samsung SSD 860 PRO 4TB
- WD Red SA500 SSD 4TB

The all disks are capable of seq. write speeds at 500 MB/s out of SATA III theorethical 600MB/s (encoded 6 Gbit/s), tested on another platforms with a working output of ~500MB/s writes (see the important note #1).

These are write speeds of R630 on Performance profile:

# dd if=/dev/zero of=/mnt/sda/test$$ bs=1M count=1000 # Micron M600
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.60953 s, 402 MB/s
# dd if=/dev/zero of=/mnt/sdb/test$$ bs=1M count=1000 # Samsung 860 PRO
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.56196 s, 409 MB/s
# dd if=/dev/zero of=/mnt/sdc/test$$ bs=1M count=1000 # WD SA500
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.6498 s, 396 MB/s

Meanwhile read speeds on all disks are normal:

# hdparm -t /dev/sda
Timing buffered disk reads: 1514 MB in 3.00 seconds = 504.11 MB/sec
# hdparm -t /dev/sdb
Timing buffered disk reads: 1618 MB in 3.00 seconds = 538.80 MB/sec
# hdparm -t /dev/sdc
Timing buffered disk reads: 1580 MB in 3.00 seconds = 526.38 MB/sec

Now what is really weird is when we change the server profile to Performance Per Watt (DAPC), the write speeds are even slower:

# dd if=/dev/zero of=/mnt/sda/test$$ bs=1M count=1000 # Micron M600
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.92744 s, 358 MB/s
# dd if=/dev/zero of=/mnt/sdb/test$$ bs=1M count=1000 # Samsung 860 PRO
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.96716 s, 353 MB/s
# dd if=/dev/zero of=/mnt/sdc/test$$ bs=1M count=1000 # WD SA500
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.0651 s, 342 MB/s

And read speeds are still fine:

# hdparm -t /dev/sda
Timing buffered disk reads: 1504 MB in 3.00 seconds = 500.95 MB/sec
# hdparm -t /dev/sdb
Timing buffered disk reads: 1612 MB in 3.00 seconds = 536.78 MB/sec
# hdparm -t /dev/sdc
Timing buffered disk reads: 1526 MB in 3.00 seconds = 508.31 MB/sec

What is on earth is going on here? Couldn't even think of a single valid reason that would cause that weirdest 50MB/s reduction. (CPU? Really?...)

All disks are at 6 Gbps:

# dmesg | grep "6.0 Gbps"
[ 2.350035] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 2.670038] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 2.994024] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
# udevadm info -q path -n /dev/sdb
/devices/pci0000:00/0000:00:11.4/ata2/host1/target1:0:0/1:0:0:0/block/sdb
# udevadm info -q path -n /dev/sdc
/devices/pci0000:00/0000:00:11.4/ata3/host2/target2:0:0/2:0:0:0/block/sdc
# lspci | grep 00:11.4
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)

I would like to hear an opinion of a trained Dell specialist on this case, and specially on the following:

1. Why the overall write speeds are being reduced up to 100MB/s on SATA ports?
2. Why the write speeds are being even more reduced (up to 150MB/s) when the profile is not Performance?
3. Based on the read speeds that are not being capped in any of the tested scenarios, is there any unidirectional artificial limit imposed on write speeds in the server's firmware?

Important note 1: It doesn't happen on the server with an exactly same chipset HP ProLiant DL360 Gen9 (E5-2600 V3 & V4) or even inferior chipset family Supermicro 1017GR-TF (E5-2600 v2), we have tested the same scenario on these servers and the SSD non-cached write speeds are constantly above 500MB/s as should be, independently of the performance profile, using same testing toolset.

Important note 2: R630 are fully capable of SATA 3.0 speeds (6Gbps / encoded 4.8Gbps), based on its technical guide located here https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/documents/dell-poweredge-r630-technical-guide-v1-6.pdf

Dell-DylanJ · Answer

Hello,

Performance tuning and specific details on why you're seeing the performance changes that you are are out of our support scope. That being said, some developers do post here from time to time, so that may be a benefit.

If you could provide details on the firmware revisions the server is running, or better than that, a SupportAssist report, I may be able to get some additional detail. I don't want to set the expectation that if you get me this report, I can definitely get you an answer, but it will better equip me to try. A performance hit to this degree seems odd to me as well.

Once you have that, you should be able to upload it here (or to a file sharing site you're comfortable with), but if you run into any issues, send me a PM. I can make sure that we find a way to get the file transferred.

timsxv · Answer

> Performance tuning and specific details on why you're seeing the performance changes that you are are out of our support scope.

We haven't made any performance tuning nor planning to do. The server configuration is a stock R630 without any changes commited running plain Debian 10, so I am not sure what is out of scope here. As for Performance word being present in the text I meant the power profile setting in BIOS [System BIOS -> System Profile Settings -> Performance or Performane Per Watt (DAPC)], it's what mostly used in the production environments.

The question here is only about stock server capabilities documented in its datasheets, without overclocking anything. Disks are capable of 500MB/s writes, server is capable of 600MB/s writes on its sata ports, so the only question here is why it's underperforming writes at 400MB/s on normal settings and below when power profile is changed to DAPC (350MB/s), while keeping reads at 500MB/s in any case.

Here is a screenshot containing the firmware versions:

Screenshot_2020-09-23_22-32-19 (copy 1).png

As you can see it's all the last versions.

However I will send you the SupportAssist data via PM shortly.

Dell-DylanJ · Answer

I'm talking with someone who is checking to see if there is an issue with the firmware to be addressed. He's in need of a bit more information.

Requested information:

- Are any tools aside from dd and hdparm being used to indicate a performance change?

- Are the drives in the server Dell branded drives? If so, part numbers would be helpful. Pictures would be even better.

- Can you confirm that the S130 controller is what you are using to control your storage?

You're welcome to send this to me as a direct message, if that would be preferable.

timsxv · Answer

Hello,

> Are any tools aside from dd and hdparm being used to indicate a performance change?

No, because these are most convenient to determine sequential speeds without interference. The only important consideration is to have zero side load on the tested drives in order to avoid any deviations in the results.

> Are the drives in the server Dell branded drives? If so, part numbers would be helpful. Pictures would be even better.

No. The models are MTFDDAK256MBF, MZ-76P4T0B and WDS400T1R0A (Links to the .pdf datasheets on their official manufacturer sites). In fact the Micron drive was pulled from Dell Latitude Rugged 5404 which was original drive for that laptop, in order to increase the variety to the test subjects. The other drives are brand new. However, the performed test showed that none of those factors affect the tests results.

> Can you confirm that the S130 controller is what you are using to control your storage?

Yes, it's S130 in the non-RAID mode (pass-through) mode. BIOS SATA mode is set to AHCI.

Dell-DylanJ · Answer

Hello,

I heard back from the escalation resource I was working with and I wanted to share his response in it's totality.

"Dell Enterprise level hard drives have firmware in place optimized to allow the highest speeds possible while ensuring stability. The hard drives in question, in conjunction with the testing methodology may not present the best barometer for the use case in this instance. The spec sheet you are referring to was created using Dell's best practices in a controlled environment with the enterprise level hardware.
If this is an issue which you are finding is impacting production performance, please open a case using the applicable service tag to allow support to assist where possible."

timsxv · Answer

Thank you. Actually, we used Dell enterprise SSDs in the past and in practice there is no notable difference between them and those SSDs except the price, also there is no issue with the testing technique as it's the most widely used.

However, today we had the opportunity to test them on Windows 2012 R2 and we were not able to reproduce the same issue on that OS - to our surprise the write speeds were just fine using the same testing techniques (sequential write with a block size of 1M). Surprisingly, this issue is now Linux related only, which is quite unusual. I will submit a bug report to the Linux kernel.

DELL-Joey C · Answer

Hey,   Thanks for the finding that it's OS specific that is causing the read write speeds. Do let us know if there is any outcome reply from Linux, so it's a share between everyone in the forum.

bsdgo · Answer

Hi @timsxv have you submitted the kernel bug and could you share the link with us? I recently had a similar issue with a X99 platform. The total speed for all sata HDDs in write mode is capped at 1250MB/s https://www.win-raid.com/t25f23-Which-are-the-quot-best-quot-Intel-AHCI-RAID-drivers-56.html I don't know if it's the chipset issue or linux kernel issue yet.

timsxv · Answer

Hello @bsdgo

I don't think the information I provided here was enough for kernel bug submission and unfortunately haven't found an opportunity to gather more information in order to make a consistent report. Also, until now I haven't found anyone else to confirm this issue, and since now I am not the only person who was able to reproduce it, I think we could cooperate on this and submit a report. I've sent you a private message.

bsdgo · Answer

Hi today I had some time and tested it on Windows using fio. The result is the total speed of all the 10 SATA ports are still limited to around 1000MB/s and can't reach 2000MB/s. So my problem appears to be related to hardware not linux or Windows specific.

I have posted on the intel forum trying to ask...

https://community.intel.com/t5/Processors/Is-there-any-total-speed-limit-for-the-X99-C610-chipset-s-ten/m-p/1241011/thread-id/48632/highlight/false

PowerEdge HDD/SCSI/RAID

R630 - artificial disk write speed limits on SATA

Was this post helpful?