I'd go for the meta set up. So create 2 RG's with 2 LUN's in each and expand the LUN's as follows: RG1 has LUN1 and LUN2 RG2 has LUN3 and LUN4 Expand LUN1 from RG1 with LUN3 from RG2 to form Meta1 Expand LUN4 from RG2 with LUN2 from RG1 to form Meta4
Why ? The beginning of each LUN is now on different RG's. It doesn't matter that much, but all bits help. For example if you have a filesystem on Meta1, it's directory table is in RG1 and a filesystem in Meta4 has the directory table on RG2. Just to spread the load a bit more. It's possibly not entirely true with new filesystems nowadays, but all bits help. This is how I'd do it.
Did I answer your questions correct ? Please mark them as such for people who will read this....
My main question is, can you have a metalun with access to too many spindles (over 3 raid groups)? But in addition, does anything else seem out of place here?
Too many spindels ? You can never have too many spindels. Metas can easily consist of LUN's coming from 3 RG's (or 6 for that matter).
We just recently got Navi Analyzer too so I'm learning as much about that as possible.
Very good
Here is our proposed setup:
DAE0 - 5x15k drives for FLARE and Reserved LUN pool (we only have a couple of snaps) DAE0 - 9x15k R5 for various purposes, mainly some SQL (but not high ultra high use) and Rational DAE1- 15x7.2k SATA - going to be file storage and low priority VMs... not critical at this point DAE2- mostly empy with 4 sata disks for expansion later DAE3- 14x15k - Going to be new VM environment
Remember to implement hot spares for each sort of drive, so a large enough FC 15k drive for the FC's and a large enough SATA 7k2 drive for the SATA's.
I was planning on doing (4+1 R5)(2+2 R10)(4+1 R5)(HS) for the new DAE(3). In addition I was going to create metaluns between the 2 R5 RGs on DAE3 and the 1 R5 RG on DAE0 to give me metaluns with access to 19 spindles.
Uhm, you only have 14 spindels in DAE3, so your setup is impossible, since your planned setup has a need of 15 spindels. The meta you proposed looks good ! A single RG can not exceed 16 spindels, but a meta can (using various RG's).
Is that going to cause me problem having that many applications having access to that many spindles at the same time? Would it be better to separate the environments by having the VMs use the two RGs on DAE3 and the SQL and other stuff use the RG on DAE0?
It depends: what are your IOps and capacity needs ? Apps with high IOps need more spindels (15k spindels can "do" about 170 IOps, so do your math). If you need dedicated performance for a certain LUN and the IOps calculation measures up, I'd say use a dedicated LUN for that application, otherwise: if you don't have a need for RDM (Raw Device Mappings), using large meta's sounds ok with me.
I do have one hot spare for each type, I just left it out of the post for some reason. Except for the mostly empty DAE, the other 3 have 15 spindles. Sorry for the confusion. My question is basically answered, but I had a follow up question:
Let's say I have two 4+1 R5 RGs. I could a) use metaluns so that LUNs have more spindles available or b) keep separate LUNs so they have less but dedicated spindles.
I realize that this impossible to answer definitively unless you know exactly how the data performs but in a general sense I like to have more spindles available for bursts of data. For instance if I have enough IOPs available in 5 spindles under normal conditions, is it still worth using a metalun to give it access to 10 spindles in case of bursty loads? Obviously this is a benefit but the drawback is the contention between the applications that now share those spindles.
Should I be more worried about the bursts of data or the peak times or more concerned about contention? Or is this impossible to answer without exact data?
I follow the same principle as you are describing. I make large "pools" of disk for servers with same types of IO patterns. My most common SAN attached host is a SQL server, so I have large pools of spindles for SQL data and another pool for SQL logs. I share the data and log pools with all of my misc. SQL servers. For my heavy-hitting servers, I dedicate spindles.
I recently implemented EMC Invista to help simplify provisioning of storage in these "pools". I create R5 3+1 groups vertically across 4 DAEs so I use all 4 buses. I then create MetaLUNS horizontally across all these spindles. Then I present these <=2TB MetaLUNS to Invista and us it to carve up virtual volumes that I present to my hosts. So, every time I create a new virtual volume (Invista term for a LUN) it's spread across 60 spindles. This is useful to me because it take a while to create a MetaLUN with 15 components, even with some of the NaviCLI scripting I have come up with.
When talking to performance experts, they typically recommend using dedicated spindles, but I have had great luck with large MetaLUNS to absorb the bursts. It also seems like an easier way to manage the storage to me (even without Invista).
Thank you both for the replies. I have already set the drives/RGs up as mentioned earlier and it seems I am on the right path.
I don't think we really have enough DAE to separate traffic properly because we have so many different types of traffic attached to the SAN (out of necessity - it's not going to change any time soon).
I went with the many spindle approach because I noticed that we have really bursty traffic. If I would allocate enough disk to cover the IO for our bursts of traffic we would have to overallocate disks for each application.
I realize this isn't 100% optimal, but then neither is using 300GB drives when you can use smaller drives (assuming the LUNs span more than one drive)... It also comes into play when deciding between 1/0 and R5. Ideally we would use 1/0 for all our VMware luns because they are very much write intensive. But for cost reasons I can't make it work for every LUN (just our most performance intesive VMs). We would need another DAE and be at full capacity with additional cost.
We just had to sacrifice a little performance for money reasons. We did find out the hard way though that using VMware (except for low priority and testing) on SATA disks just doesn't cut it...
I hear ya. I sometimes go with R1_0 over R5 even when I know R5 would be sufficient, just to keep me from having so much wasted space. If there isn't any space available on that raid group, I won't be tempted to steal it for another LUN at a later date. Sounds silly, I know, but when you need space you start stealing from whereever and there are certain raid groups I want to force myself to not steal from.
RRR
4 Operator
•
5.7K Posts
0
August 30th, 2008 04:00
I'd go for the meta set up. So create 2 RG's with 2 LUN's in each and expand the LUN's as follows:
RG1 has LUN1 and LUN2
RG2 has LUN3 and LUN4
Expand LUN1 from RG1 with LUN3 from RG2 to form Meta1
Expand LUN4 from RG2 with LUN2 from RG1 to form Meta4
Why ? The beginning of each LUN is now on different RG's. It doesn't matter that much, but all bits help. For example if you have a filesystem on Meta1, it's directory table is in RG1 and a filesystem in Meta4 has the directory table on RG2. Just to spread the load a bit more. It's possibly not entirely true with new filesystems nowadays, but all bits help. This is how I'd do it.
Did I answer your questions correct ? Please mark them as such for people who will read this....
RRR
4 Operator
•
5.7K Posts
0
August 14th, 2008 00:00
Too many spindels ? You can never have too many spindels. Metas can easily consist of LUN's coming from 3 RG's (or 6 for that matter).
Very good
DAE0 - 5x15k drives for FLARE and Reserved LUN pool (we only have a couple of snaps)
DAE0 - 9x15k R5 for various purposes, mainly some SQL (but not high ultra high use) and Rational
DAE1- 15x7.2k SATA - going to be file storage and low priority VMs... not critical at this point
DAE2- mostly empy with 4 sata disks for expansion later
DAE3- 14x15k - Going to be new VM environment
Remember to implement hot spares for each sort of drive, so a large enough FC 15k drive for the FC's and a large enough SATA 7k2 drive for the SATA's.
Uhm, you only have 14 spindels in DAE3, so your setup is impossible, since your planned setup has a need of 15 spindels.
The meta you proposed looks good ! A single RG can not exceed 16 spindels, but a meta can (using various RG's).
It depends: what are your IOps and capacity needs ? Apps with high IOps need more spindels (15k spindels can "do" about 170 IOps, so do your math). If you need dedicated performance for a certain LUN and the IOps calculation measures up, I'd say use a dedicated LUN for that application, otherwise: if you don't have a need for RDM (Raw Device Mappings), using large meta's sounds ok with me.
bpierfy
5 Posts
0
August 14th, 2008 08:00
I do have one hot spare for each type, I just left it out of the post for some reason. Except for the mostly empty DAE, the other 3 have 15 spindles. Sorry for the confusion. My question is basically answered, but I had a follow up question:
Let's say I have two 4+1 R5 RGs. I could a) use metaluns so that LUNs have more spindles available or b) keep separate LUNs so they have less but dedicated spindles.
I realize that this impossible to answer definitively unless you know exactly how the data performs but in a general sense I like to have more spindles available for bursts of data. For instance if I have enough IOPs available in 5 spindles under normal conditions, is it still worth using a metalun to give it access to 10 spindles in case of bursty loads? Obviously this is a benefit but the drawback is the contention between the applications that now share those spindles.
Should I be more worried about the bursts of data or the peak times or more concerned about contention? Or is this impossible to answer without exact data?
shewitt1
45 Posts
1
September 2nd, 2008 09:00
I recently implemented EMC Invista to help simplify provisioning of storage in these "pools". I create R5 3+1 groups vertically across 4 DAEs so I use all 4 buses. I then create MetaLUNS horizontally across all these spindles. Then I present these <=2TB MetaLUNS to Invista and us it to carve up virtual volumes that I present to my hosts. So, every time I create a new virtual volume (Invista term for a LUN) it's spread across 60 spindles. This is useful to me because it take a while to create a MetaLUN with 15 components, even with some of the NaviCLI scripting I have come up with.
When talking to performance experts, they typically recommend using dedicated spindles, but I have had great luck with large MetaLUNS to absorb the bursts. It also seems like an easier way to manage the storage to me (even without Invista).
Hope this helps
bpierfy
5 Posts
0
September 3rd, 2008 06:00
I don't think we really have enough DAE to separate traffic properly because we have so many different types of traffic attached to the SAN (out of necessity - it's not going to change any time soon).
I went with the many spindle approach because I noticed that we have really bursty traffic. If I would allocate enough disk to cover the IO for our bursts of traffic we would have to overallocate disks for each application.
I realize this isn't 100% optimal, but then neither is using 300GB drives when you can use smaller drives (assuming the LUNs span more than one drive)... It also comes into play when deciding between 1/0 and R5. Ideally we would use 1/0 for all our VMware luns because they are very much write intensive. But for cost reasons I can't make it work for every LUN (just our most performance intesive VMs). We would need another DAE and be at full capacity with additional cost.
We just had to sacrifice a little performance for money reasons. We did find out the hard way though that using VMware (except for low priority and testing) on SATA disks just doesn't cut it...
Thanks again.
shewitt1
45 Posts
0
September 3rd, 2008 20:00