VNX Storage Pool with NL-SAS in R1/0

Question

We have a VNX 5500 and trying to get it running like a well oiled machine. Our primary concern is about performance with less of a concern for capacity. We've had issues with FAST-VP with our dynamic environment workload in the past on an NS-480 we're migrating from. Since we can do mixed RAID types in FAST-VP storage pools on the VNX, I'm wondering if it's worth exploring the idea of using 2TB NL-SAS we have in R1/0 instead of the "best practice" R6 for the capacity tier. The performance tier will be comprised of 15K 600GB SAS in R5. We also have 500GB of FAST Cache but our current utilization is under 10% hit ratio.

We don't have many disks to experiment with the real workload before going past the point of no return, any real world experiences out there will be helpful to determine if this is the right approach to work around the write penalty in the slower area of the pool if it were R6 and avoid the long rebuild time vulnerability since it's R1/0.

umichklewis · Answer

You raise a very good point about RAID6 performance on VNX. While it's true that RAID6 write penalty diminishes the total number of write I/Os you can sustain, you are also limited by the small number of I/Os 7200RPM disks can deliver.

At one customer site, they experiment with a RAID10 4+4 set on 1TB NL-SAS on a VNX 5300. 8k Write IOPs were a little higher than a RAID6 12+2 NL-SAS set by about 12% percent, but Read IOPs were about 18% slower than the RAID6 set. Why? Presumably because there were nearly half the number of drives in the RAID10 set (8 in RAID10 vs. 14 in RAID6). We didn't have enough drives to create the RAID10 set with the same number of drives, so that would be the only apples-to-apples comparison you could make. So, if you're able to create a large-enough RAID10 set, you might be satisfied with the Read performance and might forgo RAID6.

In my personal opinion, however, if this will be the lowest tier for your File data (you mentioned an NS-480, so I'm assuming you're moving File data), I don't think that tier will be "busy enough" to be slowed by RAID6 nor benefit from RAID10. I'd stick with RAID6, unless you really, really have the I/O skew to justify needing a "faster" capacity tier.

Let us know what you choose and how it turns out!

Karl

kelleg · Answer

Also be aware that R6 is recommended for NL-SAS for reasons of reliability. With very large disks, if one fails and the rebuild starts, rebuilding on 7200 RPM disks takes longer than on 10K or 15K disks and the chances of a second disk failure is higher. With R6, you have better protection in case of a second fault.

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award points to the person providing the answer. This helps others searching for a similar issue.

glen

kjstech · Answer

Huh? AFAIK RAID-10 provides the absolute best protection and performance. If you have a 16 disk raid10, you have 8 disks mirroring 8 disks. It would be pretty bad if 9 disks all failed before the array could be rebuilt.

Or even an 8 disk R10, if 5 disks failed, thats a lot of failures.

On Raid 6, only 2 drives are parity so if a third drive fails your SOL.

On Raid 5 only 1 drive is lost to parity so no one should ever consider Raid 5 with the size of disks these days.

dynamox · Answer

loose two disk in RAID10 that are mirror pairs and entire RG is toast.

I am sure your EMC account team loves how you think that RAID-5 should never be considered. Raid-5 is the most economical choice given it's performance and redundancy. How many times have you lost 2 disk in a RAID-5 set (in an enterprise level array now, i don't care about a crappy Buffalo "wanna be" array) ? I've been using EMC storage going back to Clariion FC4700 (early 2000), i have only had one incident where we lost a RAID-5 raid group (NS80). Agree, drives are getting bigger, people chose to go to RAID-6 (and incur 6x write penalty in addition to an additional parity) and/or be conservative with RAID-5 RG ..keep them at 4+1R5 instead, smaller RG = faster rebuilds = lesser windows for dual drive failure.

kjstech · Answer

I see where your coming from. Just a lot of people have been steering me away from R5, not just the sales guys, but other I.T. and storage professionals. I guess the thought is these days with the larger disk sizes, the rebuild time is far more costly in resources and time, so its quite possible to lose another disk during that rebuild. Ok you lost one RAID-5 in the last 14 years and to me thats 1 raid too many in an enterprise situation.

I struggle with this decision every single day (as my new VNX isn't up yet - doing some power work first) as to which raid I want to go with ... 10 or 6. Our sales engineer acknowledges the write penalty but based on our NX4 data logging he looked through he said with Fast CACHE (we have 3 x 200GB SSD's) it would help negate that write penalty and we would still be 1000 IOPS higher than our 95th percentile on our NX4 today. He said we could go either way, 10 or 6... depends on how many IOPS we require.

The only thing I did not get clarification on is the number of spindles. We have 17 x 2 TB NL-SAS and 17 x 600 GB SAS. One of each will be hot spare. So now the big decision is more spindles for more IOPS ie) 16 disk raid groups, one in each pool, or two raid groups in each pool of 8 disks each. Wheather its 8 disks or 16, both work with RAID6 or RAID10, so now the last factor is deciding how much space do we need. We would have enough TODAY if we went RAID10, but that doesn't account into the future. I tend to lean towards 10, but having half the capacity is a major turnoff. However the NX4 at our DR site does not have enough capacity if we went RAID6, and we want to replicate to it, so we may just go RAID10 the whole way out, though I can move some DAE's from our NX4 at production site to DR site.... that is another discussion for another thread. I'm leaning towards RAID10 for the SAS drives and RAID6 for the NL-SAS and letting Fast VP figure it out.

I've just seen a lot of stories over at Spiceworks forums (and others) plus posts from greats like Scott Alan Miller all pushing towards RAID10, with good reasons. As you add drives to RAID 5/6 the array gets larger and the risk goes up rapidly as losing 1 out of 3 drives is very different than losing 1 out of 8 drives.

Anonymous User · Answer

With the amount and size of drives you have I would R5 the 600GB drives R6 the 2TB drives and ensure that you monitor your applications and then add the busy ones into the Fast Cache environment as needed.

R1/0 is a total waste of your resources in the way of disk.

My 2c worth.

dynamox · Answer

To give you some food for thought . After our dual-drive incident we talked to a statistician at EMC, crazy smart guy who's sole purpose in life is to crunch numbers for drive failure probability. So for an NS80 (CX3 backend), with ~500 drives (mostly 1TB, raid-5 4+1) ..probability of losing 2 drives in one raid group was 1 in 147 years. 'Lucky' for us, it was 2 years ago when we&#xa0; fell somewhere in that range and we lost data.

kjstech · Answer

But going R5, I think you would be talking 3 x 5 disk raid groups and then two hot spares right?

The concern is Raid 5 can only absorb one drive failure. Raid 6 has a hard stop of two drive failures. Raid 10 will probably absorb two fails, likely aborb three fails and might absorb four or more if its a large enough array.

But if you think the VNX has enough enterprise proactive failure prediction and fast rebuild times, then maybe ONLY because its an EMC VNX, Raid 5 is ok.

With all the marketing acronyms and features, its interesting no one came up with a RAID-6 ASIC that negated the 6:1 write penalty. That would make RAID 6 the clear choice for me. Some seem to think the Fast Cache will help absorb that delay between the vm hosts and the shared storage though. Will it? I guess I would have to test that in our specific environment.

Anonymous User · Answer

As Dynamox has pointed out, the chances of a double drive failure (DDF) on a raid set are slim. But alas the bigger our storage systems get the better the probability of one happening.

As an EMC field engineer I was in the field swapping drives (Amongst many other duties) for a good ten years.

In all that time I only had to attend one DDF and I dealt with a LOT of customers. This failure was on a 9 storage bay DMX using R5 and we where lucky in this case that the lab intervened in time to prevent data lose. (So its not all bad news :-))

The larger the system and the more drives leads to bigger risk of DDF. So Symmetrix size systems are at much more risk than the normal VNX size systems. We also need to factor in that EMC do a lot of pro-active drive swaps which helps mitigate DDF. I stand by my original recommendation R5 for 600Gb. You can use R5 4+1 OR R5 8+1 depending on the number of disks you have and your IOPs requirement. Using R6 6+2 or 14+2 for the large 2TB drives which will require a larger re-build time.

When it comes to HotSpares 1 per 30 drives should be fine for most environments. You can spare a 600GB disk to a larger disk but obviously not the other way round. If you do spare fast disk to a large slow disk performance of the raid set will drop to that of the larger disk. (I believe)

vnx-best-practices-for-performance-whitepaper would be an excellent guide to read. Only 30 pages so not too big.

Hope this helps.

VNX

VNX Storage Pool with NL-SAS in R1/0

Was this post helpful?