This post is more than 5 years old
48 Posts
0
2276
LUN standard for ESX clusters.
Hello to the group,
We currently use NFS for ESX datastores but will be moving to VMAX block storage. I'm sure many of you have experience in this environment so I wondered if anyone would mind sharing how you manage provisioning to ESX in terms of a standard lun size.
I've been considering the idea of a 4TB standard lun created as a striped meta and for ESX I'd prefer to provision them in batches so that it only needs to occur infrequently. Idealy these would be used for individual datastores and never expanded.
What works best in your environment? Do you concatenated standard sized luns or use them as one lun per datastore? Do you allow expansion?
I've read the best practices doc and don't believe these questions are fully covered. Probably because there's more than one correct answer and it depends on the environment. I'm just looking for a starting point and examples of things that have been shown to work.
Thanks,
Vic
M_Salem
213 Posts
0
August 25th, 2014 14:00
You are most welcome Vic. So let us talk first about SYMM MAXIMUMs here, So for SYMM VMAX series the max standalone device is 240 GB and if you want to create METAs you have a limit of 255 members so the max size you can create out of a SYMM device is almost 60 TB. This has been tested and approved by EMC labs. However, I will not create a single datstore of this size for many reasons, If i want to replicate using SRM/SRA, The remote array may not have enough disk space to create 255 dev of size 240 GB to create the RDF pair, Another thing, comes when you have IO intensive machine, This MAY dramatically affect the performance of other VMs, Add to this the LUN queuing at VMKERNEL level etc...(This can be changed if recommended by vmware though!).
So my advice is to use, 2 or 4TB datastores and do not concatenate on the ESX level, Just create 8-way META with member size of 240 GBs (approx. 1.88 TB) or 16-way META for size of 240 GBs each member (almost 4 TB)
2 or 4 TB should work with almost 95-99 % of your workload, If you have a machine that needs more than that, Then use the new storage enhancements like jumbo vmdk (with vmware esxi 5.5) on a new sperate datastore or so,
When you want to expand in the future, For Concatentaed METAs just add a new device of same configuration to the META head and expand from ESXi host. For Striped META, using protect_data option you can expand but you will need to create a BCV device to with the same configuration as the current Striped META.
I will not recommend using extents/chunks of datastores from ESXi level. Losing one of these may corrupt the whole infrastructire due to the fact that vmkernel stripes the writes across all extents so loss of one is like loss of all. VMware support have very limited options to recover the datastore.
You can leverge Datastores clusters. It is areally nice feature but make sure you combine datastores of same technology to have a reasonable recommendations by SDRS.
So combine 2x4 TBs datstores into one Cluster and let SDRS make decisions.
A really good & recommended techbook document was written by Drew and Cody, You can find it in the below link:
https://support.emc.com/docu33910_TechBook:-Using-EMC-Symmetrix-Storage-in-VMware-vSphere-Environments.pdf?language=en_US
Hope this helps
Mohammed Salem
M_Salem
213 Posts
1
August 25th, 2014 08:00
Hi Vic,
Welcome to EMC Community.
I would recommend using striped meta volumes as you mentioned even with thin provisioning. For the standard size, that depends on many factors, VMDK sizes, vCPUs and RAM assigned to your VMs, IO intensive VMs. while large datastores might be good to manage in terms of smaller numbers are presented to the ESXi farm, It may lead to LUN queuing issues, Thanks to VAAI ATS that is supported with SYMM and VNX, There is no locking issues like we used to have in the past. 4 TB is good for newly presented Volumes formatted with VMFS5. if you have an existing VMFS3, you can upgrade to VMFS5 online and it will take most (not all as it is called upgraded vmfs5) features of VMFS5.
Another thing, 4TB was considered a very good option till esxi5.1 but with esxi5.5 Jumbo VMDK were introduced so there is no limitation of 2 TB vmdk anymore and i have seen some customers using Jumbo VMDKs instead of using volume managers on the gues OS side so this is something that you may consider.
Other things to consider, Do you have vCD, VDI or vCAC implemented, Those usually consumes space. Are you using Snapshots heavily? That is another question that may make you reconsider the datastore sizes.
You can make use of Datastores Clusters and use it with SDRS, Many customer are using this nowadays and SYMM supports this even with FAST VP enabled on the array level (however SDRS movements should be based on capacity not IO latency).
Those are the factors i usually consider before deciding to go for new storage provisioing in any VMware environment of my customers. The requirements differ from customer to another but usually the minimum i have seen since esx4.1 was 2 TB, Now we are going above and beyond this.
Hope that helps
Mohammed Salem
vic_engle
48 Posts
0
August 25th, 2014 13:00
Thanks for the input Mohammed. A couple of follow up questions. Do your customers concatenate LUNs to form larger datastores? What is the largest striped meta size you're creating for customers?
When considering a large striped meta, I was thinking more about practicality within the VMAX. I know the latest version of ESX supports very large LUNs but most things I've read advise against really big LUNs. On the VMAX I have to work within the 240GB volume size limit for creating meta members and for a striped meta, I don't want to add more than 16 members if possible. Certainly no more than 32 members. And on the ESX side I understand there are some host side disk I/O queuing considerations that work against the idea of really large luns.
So, I'm just trying to get an idea about how people typically manage ESX storage from a VMAX array.
vic_engle
48 Posts
0
August 25th, 2014 18:00
Thanks again Mohammed!
alcon483
2 Posts
0
October 10th, 2014 07:00
Hi Mohammed,
I havebeen following the standard of 2&4TB in our environment and it works well. Now we are thinking about doing 16TB so what is your throughts as far as Meta memeber size for 16TB?
Thanks,
GK
uid_0
4 Posts
0
October 10th, 2014 07:00
For our VMware environment - usually we just use the max meta member size of 262668 CYL and form a striped meta to get the nearest size we need.
For the most part - Our VMware environment uses 4088GB VMFS vols (17 - 262668 CYL meta members.) We used to try to keep the meta members in powers of 2, but we started deviating from that once TDEVs became pretty much the norm.
We didn't want to get super small with the meta member size because we've been hit with the 4096 dev limit before and opted for larger members, rather than smaller.
Using 262668 CYL members - that essentially gives us 960TB we could present to each port group. We have plenty of wiggle room to work with.
We haven't had any issues with regard to performance, we we don't have any issues with regard to SRDF/A (R1 and R2 side) either.
Of course - we have some circumstances that dictate smaller meta members - but we try to use 262668 CYL members whenever we can, especially with large VMFS volumes.
alcon483
2 Posts
0
October 10th, 2014 08:00
so the Max meta member size is 262668cyl. ?
pappc
52 Posts
0
October 15th, 2014 06:00
Effectively, yes the maximum size a Meta member can be is 262668 cylinders, as that is the maximum size of a Symmetrix device in VMAX series arrays.
I would just like to point out one hard rule in the array's design with regards to Meta devices and their subsequent expansions that can be overlooked. A hard rule exists that says every member of the Meta must be the same exact size as all the other members. So, for example, if you have a Meta with existing members at 60000 cylinders each, you can only expand it with 60000 cylinder devices for the rest of its existence.
Regards,
Chris
PedalHarder
465 Posts
0
October 15th, 2014 23:00
Regarding meta members being the same size, this is certainly true for striped meta devices. It is possible to add meta members to a concatenated meta device that are unlike sizes. This is not a typical deployment for meta devices and not a recommendation, just a clarification on what is possible.