Unsolved
This post is more than 5 years old
85 Posts
0
2508
August 31st, 2012 06:00
Performance Issues with VMware Environment
We have VMware on UCS Blade and connected VMAX.
Before the VMware Production Database Cluster moved VMAX(Raid 5 3+1 thin pool) on Seperate Pool of 200 FC spindles. It is layed out on raid 10 on CX4960(Raid 10).
The VMs did nothave issues and even the backup of normal DB used to take 30 -40 minutes. After moving to VMax the application suffering the Latencies.
We have migrated the TDEV from one pool to another just verify it as the Pool is not issue here. Even then we have issues. The VM is moved to Datastore back on CX and the Performance just improved.
We are trying to figure what is that causing this kind og bottlencecks.
Does any one has any thoughts on this and feedback is much appreciated.


Govindagouda
85 Posts
0
October 12th, 2012 13:00
We have not resolved this issue. We have done every kind of test here is what we are done till now
We are still waiting for EMC to check what is going on with the frame. We are literally seeing 200 to 300 ms latency on the vmax Fiber chanel disks.
Govindagouda
85 Posts
0
October 15th, 2012 05:00
Has any one see this kind of issues on Vmware. The UCS environment Works perfectly without changes in the Environment. As soon as the Storage VMare vmotioned to VMAX datastore we see the issues .
We know this not issue with the UCS infra as the same configuration works fine with Clariion.
Quincy561
1.3K Posts
0
October 15th, 2012 06:00
VNX and VMAX have different software architectures. Did you try changing the parameter that sets how many IOs are sent down a channel before switching? Did you try more paths?
What is the workload? It is possible that the type of workload you are running is just faster on the VNX than the VMAX.
Govindagouda
85 Posts
0
October 15th, 2012 07:00
Hi Sasi,
Thanks for your email. We have Set these settings on the initiator groups.
Thanks and Regards,
Govind.N.R
Storage Engineer, Information Technology
WellCare Health Plans, Inc.
8735 Henderson Rd, Ren 1, Tampa, FL 33634 | Tel-WOrk: 813-206-5731 x5731 | BB: 8130351-9762 |email: rgovind1@wellcare.com
Govindagouda
85 Posts
0
October 15th, 2012 07:00
We are on round robin and our IO switch value is 1. Which means every 1 IO the ESX host will switch the path. This is recommended by EMC after we have the issue when the default value is 1000.
Thanks and Regards,
Govind.N.R
Storage Engineer, Information Technology
WellCare Health Plans, Inc.
8735 Henderson Rd, Ren 1, Tampa, FL 33634 | Tel-WOrk: 813-206-5731 x5731 | BB: 8130351-9762 |email: rgovind1@wellcare.com
Govindagouda
85 Posts
0
October 15th, 2012 07:00
We have 8 paths 4 per fabric to each host. Our current setting on the Host for the IO is round robin and it is set to 1 IO before it switches.
Thanks and Regards,
Govind.N.R
Storage Engineer, Information Technology
WellCare Health Plans, Inc.
8735 Henderson Rd, Ren 1, Tampa, FL 33634 | Tel-WOrk: 813-206-5731 x5731 | BB: 8130351-9762 |email: rgovind1@wellcare.com
Quincy561
1.3K Posts
0
October 15th, 2012 07:00
What is the workload? What is the performance expectations for the workload? What do you see on VNX and on VMAX? How many FAs on the VMAX?
Sasi_kumar_485147
2 Posts
0
October 15th, 2012 07:00
Hi,
Hope all required port flags are set for VMware environment,
i) Common Serial Number (C)
ii) VCM State (VCM) --- (ACLX for V-MAX)
iii) SCSI 3 (SC3)
iv) SPC 2
v) Unique World Wide Name (UWWN)
vi) Auto-negotiation (EAN)
vii) Point to Point (P)
Thanks,
Sasi
Sasi_kumar_485147
2 Posts
0
October 15th, 2012 07:00
Hi Govind,
Did you give a try using the path policy to Round Robin if it is in Fixed or MRU.
Thanks,
Sasi
Quincy561
1.3K Posts
0
October 16th, 2012 13:00
I didn't see an answer to the question about the workload profile and performance expectation.
Govindagouda
85 Posts
0
October 16th, 2012 13:00
What is that Quincy you are looking for.
We have Good Sized SQL host with 8 to 10 TB VMS with bulk data read and writes withing ESX Farm.
Our work load is like 80% read and 20% write.
Thanks
Govind
Quincy561
1.3K Posts
0
October 16th, 2012 13:00
Ok, just wondering if the workload was OLTP like, or DW like, or maybe something single threaded like file copy. From the "bulk read and write" comment, it sounds like it could be a file copy type workload. This is one where adding concurrency may help if possible.
Govindagouda
85 Posts
0
October 16th, 2012 13:00
Here is what our SQL hosts are doing
We have an APP that reads the data from one database and then make three times write on other 3 hosts with additional data that reads from files.
Our SQL Vms are located on different Meta devices on the VMax.
Thanks
govind
Quincy561
1.3K Posts
0
October 16th, 2012 13:00
And what is the performance expectation? We should see about 100-150MB/sec for a file copy on VMAX.
Govindagouda
85 Posts
0
October 16th, 2012 14:00
we are not seeing that.
Here are few of the tests we did
a) On VM those have issues. We copied data from our NAS(EMc Celerra) and the copy rate is 150 to 200 MB/s
b) We copied the data from one VMax disk to other VMax disk on the same host the Copy rate is 30 to 40 MB/s
c) We copied data from one disk to NAS the Copy rate is same 40 to 50 MB/s.
We are not understanding how the data from NAS to VMax disk is faster and not the other way. This is one of the simple test not giving us clue on the issue.
On ESX we see io latency upto 400 ms. which is not accpetable at all.