6 Posts

1192

February 21st, 2023 16:00

Maximizing RAID10 array SSD Speeds R730xd PERC H730 Mini 24 disks

Hi all,

I'm currently running an R730xd 12x 3.5" + 4x 3.5" Midplane chassis that's lined with 16x PM1643a's in a single RAID10 array on the PERC H730 Mini onboard controller.  I'm doing some minor upgrades including swapping to a 24 bay 2.5" chassis and I've been looking to eventually expand the RAID10 to 24x PM1643a's.  Based on the best research and info I could find for days, I believe I'm already hitting a sort of speed bottleneck, and the 24x disk array wont even be possible since the maximum I'm reading on this setup would be 16 disks in a single array (which I currently have).  So, I've been wondering what my options are.

The first question I have is related to the above pre-existing bottleneck I think I have.  Given the PM1643a is essentially maxing out the 12Gbps SAS interface at 1700MB/s, and since I have 16 of them, am I correct in assuming the 16 disks are sharing only 8 total SAS ports at this speed based on the H730 specs?  If so, would this mean that each SAS backplane slot is only getting 6Gbps instead of 12Gbps?  Having said that, if I filled out the array to 24 disks (if possible) it'd be docked even further to 3Gbps per SAS backplane slot?

The 2nd part is if the above slashes in speeds are indeed accurate, what are my options here?  The first thing that came to mind was adding a couple PERC cards to the PCIE slots... but this introduces other variables including if my 2x E5-2699's have enough PCI lanes for this and what this sort of setup would look like.  If I got an extra 2 PCIE PERC controllers and dedicated 8 disks per controller, is there a way to easily create a big 24x RAID10 array that's leveraging 3 total controllers?  Or would I have to do something like create 3x RAID10 arrays of 8 disks each, and then software RAID them after the fact?

If anyone could shed some light onto how to tackle this issue here it'd be greatly appreciated!  Thanks!

6 Posts

February 27th, 2023 20:00

Synopsis:

I was able to get this setup successfully.  I swapped to the 24 bay chassis and am leveraging 2 PERC H730p controllers: 1 PCIE and 1 Embedded.  The key here is leveraging the Flex Zoning feature of the 24x 2.5" backplane by setting the Split Mode option in the iDRAC.  Note: this setting must be changed in iDRAC prior to connecting the 2nd PERC.  This particular rig is on iDRAC8 Enterprise and this setting can be found at:

Overview >> Storage >> Enclosures >> Setup >> Split Mode in the pull-down menu




Jeff86b_1-1677558675057.png

This sets the 24x backplane into 12x + 12x mode where SAS connectors A0/B0 [1] are wired to the embedded PERC and connectors A1/B1 [2] are wired to the PCIE PERC:

Jeff86b_2-1677559006505.png
Note:
A0/BO = slots 0 to 11 = Embedded PERC
A1/B1 = slots 12 to 23 = PCIE PERC

After getting everything wired and plugged correctly, you can reboot into the LIfeCycle Controller and setup new Virtual Disks on each controller.  Here's what I did:  Since I'm leveraging more than 12 disks, I created 2 separate virtual disks in RAID10.  Then, I created a software RAID0 for the 2 VD's via mdadm (I'm running Ubuntu Server 22.04).  The thought process here is as long as the underlying RAID10 arrays are managed correctly if failures occur, the RAID0 should never be affected or break as a result, thus mimicking a large RAID10 array.

I can confirm I'm successfully surpassing 7.8GB/s limitations in of a single PERC controller and my benchmarks are stunning:  I'm seeing roughly a 2.5x increase in performance as a result of leveraging the split mode and running 2 controllers!

I hope this helps anyone stumbling across this same situation!







 

6 Posts

February 22nd, 2023 00:00

Adding to this after finding additional info:

It appears that the PCIE 3.0 x8 card has a limited throughput of roughly 8GB/s.  If the onboard/embedded controller holds the same, it looks like the maximum throughput we're looking at is 16GB/s.  I additionally found that the 24x backplane can be split to a 12/12 but not an 8/8/8 or 6/6/6/6 so it looks like I'll only be able to leverage 2 RAID controllers in this situation for maximization.

Moderator

 • 

5.4K Posts

 • 

37 Points

February 22nd, 2023 21:00

Hello thanks for choosing Dell. This is just my humble opinion, but I don't see amber alert, I don't see any issues here. You just have a lot of SSDs on your controller.

No Events found!

Top