Highlighted
2 Iron

LUN standard for ESX clusters.

Jump to solution


Hello to the group,

We currently use NFS for ESX datastores but will be moving to VMAX block storage. I'm sure many of you have experience in this environment so I wondered if anyone would mind sharing how you manage provisioning to ESX in terms of a standard lun size.

I've been considering the idea of a 4TB standard lun created as a striped meta and for ESX I'd prefer to provision them in batches so that it only needs to occur infrequently. Idealy these would be used for individual datastores and never expanded.

What works best in your environment? Do you concatenated standard sized luns or use them as one lun per datastore? Do you allow expansion?

I've read the best practices doc and don't believe these questions are fully covered. Probably because there's more than one correct answer and it depends on the environment. I'm just looking for a starting point and examples of things that have been shown to work.

Thanks,

Vic

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
3 Argentum

Re: LUN standard for ESX clusters.

Jump to solution

You are most welcome Vic. So let us talk first  about SYMM MAXIMUMs here, So for SYMM VMAX series the max standalone device is 240 GB and if you want to create METAs you have  a limit of 255 members so the max size you can create out of  a SYMM device is almost 60 TB. This has been tested and approved by EMC labs. However, I will not create a single datstore of this size for many reasons, If i want to replicate using SRM/SRA, The remote array may not have enough disk space to create 255 dev of size 240 GB to create the RDF pair, Another thing, comes when you have IO intensive machine, This MAY dramatically affect the performance of other VMs, Add to this the LUN queuing at VMKERNEL level etc...(This can be changed if recommended by vmware though!).

So my advice is to use, 2 or 4TB datastores and do not concatenate on the ESX level, Just create 8-way META with member size of 240 GBs (approx. 1.88 TB) or 16-way META for size of 240 GBs each member (almost 4 TB)

2 or 4 TB should work with almost 95-99 % of your workload, If you have  a machine that needs more than that, Then use the new storage enhancements like jumbo vmdk (with vmware esxi 5.5) on a new sperate datastore or so,

When you want to expand in the future, For Concatentaed METAs just add a new device of same configuration to the META head and expand from ESXi host. For Striped META, using protect_data option you can expand but you will need to create a BCV device to with the same configuration as the current Striped META.

I will not recommend using extents/chunks of datastores from ESXi level. Losing one of these may corrupt the whole infrastructire due to the fact that vmkernel stripes the writes across all extents so loss of one is like loss of all. VMware support have very limited options to recover the datastore.

You can leverge Datastores clusters. It is  areally nice feature but make sure you combine datastores of same technology to have a reasonable recommendations by SDRS.

So combine 2x4 TBs datstores into one Cluster and let SDRS make decisions.

A really good & recommended techbook document was written by Drew and Cody, You can find it in the below link:

https://support.emc.com/docu33910_TechBook:-Using-EMC-Symmetrix-Storage-in-VMware-vSphere-Environmen...

Hope this helps

Mohammed Salem

View solution in original post

0 Kudos
9 Replies
Highlighted
3 Argentum

Re: LUN standard for ESX clusters.

Jump to solution

Hi Vic,

Welcome to EMC Community.


I would recommend using striped meta volumes as you mentioned even with thin provisioning. For the standard size, that depends on many factors, VMDK sizes, vCPUs and RAM assigned to your VMs, IO intensive VMs. while large datastores might be good to manage in terms of smaller numbers are presented to the ESXi farm, It may lead to LUN queuing issues, Thanks to VAAI ATS that is supported with SYMM and VNX, There is no locking issues like we used to have in the past. 4 TB is good for newly presented Volumes formatted with VMFS5. if you have an existing VMFS3, you can upgrade to VMFS5 online and it will take most (not all as it is called upgraded vmfs5) features of VMFS5.

Another thing, 4TB was considered a very good option till esxi5.1 but with esxi5.5 Jumbo VMDK were introduced so there is no limitation of 2 TB vmdk anymore and i have seen some customers using Jumbo VMDKs instead of using volume managers on the gues OS side so this is something that you may consider.

Other things to consider, Do you have vCD, VDI or vCAC implemented, Those usually consumes space. Are you using Snapshots heavily? That is another question that may make you reconsider the datastore sizes.

You can make use of Datastores Clusters and use it with SDRS, Many customer are using this nowadays and SYMM supports this even with FAST VP enabled on the array level (however SDRS movements should be based on capacity not IO latency).

Those are the factors i usually consider before deciding to go for new storage provisioing in any VMware environment of my customers. The requirements differ from customer to another but usually the minimum i have seen since esx4.1 was 2 TB, Now we are going above and beyond this.

Hope that helps

Mohammed Salem

Highlighted
2 Iron

Re: LUN standard for ESX clusters.

Jump to solution

Thanks for the input Mohammed. A couple of follow up questions. Do your customers concatenate LUNs to form larger datastores? What is the largest striped meta size you're creating for customers?

When considering a large striped meta, I was thinking more about practicality within the VMAX. I know the latest version of ESX supports very large LUNs but most things I've read advise against really big LUNs. On the VMAX I have to work within the 240GB volume size limit for creating meta members and for a striped meta, I don't want to add more than 16 members if possible. Certainly no more than 32 members. And on the ESX side I understand there are some host side disk I/O queuing considerations that work against the idea of really large luns.

So, I'm just trying to get an idea about how people typically manage ESX storage from a VMAX array.

0 Kudos
Highlighted
3 Argentum

Re: LUN standard for ESX clusters.

Jump to solution

You are most welcome Vic. So let us talk first  about SYMM MAXIMUMs here, So for SYMM VMAX series the max standalone device is 240 GB and if you want to create METAs you have  a limit of 255 members so the max size you can create out of  a SYMM device is almost 60 TB. This has been tested and approved by EMC labs. However, I will not create a single datstore of this size for many reasons, If i want to replicate using SRM/SRA, The remote array may not have enough disk space to create 255 dev of size 240 GB to create the RDF pair, Another thing, comes when you have IO intensive machine, This MAY dramatically affect the performance of other VMs, Add to this the LUN queuing at VMKERNEL level etc...(This can be changed if recommended by vmware though!).

So my advice is to use, 2 or 4TB datastores and do not concatenate on the ESX level, Just create 8-way META with member size of 240 GBs (approx. 1.88 TB) or 16-way META for size of 240 GBs each member (almost 4 TB)

2 or 4 TB should work with almost 95-99 % of your workload, If you have  a machine that needs more than that, Then use the new storage enhancements like jumbo vmdk (with vmware esxi 5.5) on a new sperate datastore or so,

When you want to expand in the future, For Concatentaed METAs just add a new device of same configuration to the META head and expand from ESXi host. For Striped META, using protect_data option you can expand but you will need to create a BCV device to with the same configuration as the current Striped META.

I will not recommend using extents/chunks of datastores from ESXi level. Losing one of these may corrupt the whole infrastructire due to the fact that vmkernel stripes the writes across all extents so loss of one is like loss of all. VMware support have very limited options to recover the datastore.

You can leverge Datastores clusters. It is  areally nice feature but make sure you combine datastores of same technology to have a reasonable recommendations by SDRS.

So combine 2x4 TBs datstores into one Cluster and let SDRS make decisions.

A really good & recommended techbook document was written by Drew and Cody, You can find it in the below link:

https://support.emc.com/docu33910_TechBook:-Using-EMC-Symmetrix-Storage-in-VMware-vSphere-Environmen...

Hope this helps

Mohammed Salem

View solution in original post

0 Kudos
Highlighted
2 Iron

Re: LUN standard for ESX clusters.

Jump to solution

Thanks again Mohammed!

0 Kudos
Highlighted
Not applicable

Re: LUN standard for ESX clusters.

Jump to solution

Hi Mohammed,

I havebeen following the standard of  2&4TB in our environment and it works well. Now we are thinking about doing 16TB so what is your throughts as far as Meta memeber size for 16TB?

Thanks,

GK

0 Kudos
Highlighted
2 Bronze

Re: LUN standard for ESX clusters.

Jump to solution

For our VMware environment - usually we just use the max meta member size of 262668 CYL and form a striped meta to get the nearest size we need.

For the most part - Our VMware environment uses 4088GB VMFS vols (17 - 262668 CYL meta members.)  We used to try to keep the meta members in powers of 2, but we started deviating from that once TDEVs became pretty much the norm.

We didn't want to get super small with the meta member size because we've been hit with the 4096 dev limit before and opted for larger members, rather than smaller. 

Using 262668 CYL members - that essentially gives us 960TB we could present to each port group.  We have plenty of wiggle room to work with.

We haven't had any issues with regard to performance, we we don't have any issues with regard to SRDF/A (R1 and R2 side) either.

Of course - we have some circumstances that dictate smaller meta members - but we try to use 262668 CYL members whenever we can, especially with large VMFS volumes.

0 Kudos
Highlighted
Not applicable

Re: LUN standard for ESX clusters.

Jump to solution

so the Max meta member size is  262668cyl.  ?

0 Kudos
Highlighted
3 Silver

Re: LUN standard for ESX clusters.

Jump to solution

Effectively, yes the maximum size a Meta member can be is 262668 cylinders, as that is the maximum size of a Symmetrix device in VMAX series arrays.

I would just like to point out one hard rule in the array's design with regards to Meta devices and their subsequent expansions that can be overlooked.  A hard rule exists that says every member of the Meta must be the same exact size as all the other members.  So, for example, if you have a Meta with existing members at 60000 cylinders each, you can only expand it with 60000 cylinder devices for the rest of its existence.


Regards,

Chris

0 Kudos
Highlighted
3 Zinc

Re: LUN standard for ESX clusters.

Jump to solution

Regarding meta members being the same size, this is certainly true for striped meta devices. It is possible to add meta members to a concatenated meta device that are unlike sizes. This is not a typical deployment for meta devices and not a recommendation, just a clarification on what is possible.

0 Kudos