VNX7500 Array - Citrix PVS 6.5 sluggish performance

i tried to open the nar file and it said incorrect archive version, odd ..tried it from VNX running 32p11.

I checked for you, but I can open both files normally on a VNX5700 running 32.011. A quick look doesn't really indicate a problem on the box. For example lun 16 is relatively busy, but with low responsetimes (<6ms).

But only spent a couple of minutes, did not dig deep...

Have you tried with Unisphere Client to open the file locally on your laptop/desktop?

Can you specify any time frame that we need to check to find any anomalies?

However, there is one general observation.

Pool 2 has 27 disks

Pool 8 has 40 disks

All other pools have 30 disks

All the pools are RAID5, so Pool 2 has 27 disks, which is not divisible by 5. This means that one/two of the private RGs created in a pool will have less than 5 disks. This will give a lower performance characteristic to these particular private RGs and hence the lun extents residing on them. But in the same pool, for the other lun extents, the performance will be fine. Hence this may explain the unpredictable/random nature of the performance issue.

did you look at spindles of pool 2 and saw IOPs that exceed "documented" values ?

Hi Dynamox, I checked the NAR files, but the disks on Pool 2 in both the NAR files seem to be running below their hardware specification values. They are 300GB SAS drives but not sure if they are 15/10K RPM. I may have been mistaken, can you please help me confirm?

i am not sure why but still can't get the NAR file to open, even though i am connected to VNX running 32 p11. Will need to update my local Uni client/server installation like you did and try again locally.

Hi Dynamox.

Do you want me to arrange a file transfer for these NAR files. please email me on firoz_ali@jltgroup.com incase you can do a webex transfer or Teamviewer transfer.

regards

firoz

are you seeing the error ''The system could not be verify the certificate''?

Hi Sushant

Can you also put some lights on the RLP luns which is POOL 4 (RAID 10). when I use the search feature in the analyzer to find the busiest luns i find major luns from Pool 4 which are RLP. Now i can't see if my RLPs are too contributing to SP forced flush and or if they are causing issues for the Citrix Xenapp luns which I noted in my original description.

regards

Hi Sushant

All the drives used are 15k RPM. (both 300 GB and 600 GB)

I can get to your file ok but for some reason I can’t open the NAR file. I even updated to the latest Unisphere Client/Server version today and still could not open it. Others have opened it so much be something on my side.

Hi All

Apologies for not responding during the last two days as I was busy at DC doing fiber cablings.

I have moved some RAID10 Luns from RG 22, 23, 24 which were causing SP forced flushing  to a 32 drives RG (RAID 10).

I did it tonight ie.7th Feb and hope to see some better memory utilization tomorrow i.e. 8th Feb.

------------------------------------------------------------------------------------------------------------------------------------

Update.

We did a POC by creating a fresh 2 x 16 drives RG of RAID 10. Striped a metalun of 300 GB across both the newly created RG and vmotion'd a Xenapp VM. Added only 10 users to see if they see performance problems but to my expectation they did face the same problem. I was pretty confident that the problem lies somewhere else. As sushant noted about pool 2 drives sets and the lun 16 which are quite busy but this cannot cause problems to a large estate of 800 users. Some memory spikes are too seen but they are just once or twice during a day which can be ruled out.

Let me know if theirs any other surprises in the NAR files which I couldnt notified.

appreciate sushants efforts

regards

Hi Sushant

The time frame you can view is from 8:00 hrs to 17:59 hrs GMT.

thanks for the Pool 2 details you provided but i have a question for you. Now that private RG uses 4+1 groups which means 5 drives each so I am sure till the drive 25 i have 5 pvt RGs in it. Now the last 2 drives can I make to 4+1 by adding another 3 drives?

Curiously can't open the first attachment but no worries as the merged one loads just fine with the latest Unisphere client and server (Dynamox is this the NAR file you reported about earlier?).

While I'm sitting here waiting on copies to finish, I thought I'd do a quick review and simply dump the items that stood out without necessarily correlating or judging the individual items.  Maybe this list would provide ideas for others that might be able to spend more time on it.  I will also mention that in cases like this, we also ask for SP Collects.  Can I assume that you also have a ticket open with support to review?

Again, the following is just a dump of those items that stood out like the proverbial sore thumb:

1) Pool 2 FAST Cache is disabled

2) Pool 2 has a private RAID group with a 2+1 construct (as well as a 5+1)

- I'm less concerned about the fact there is a 5+1 (or in other words anything that doesn't grossly deviate from the rest or the recommended multiples)

To answer your question, once the private RG's are created they are immutable.  Expanding with new drives are used to build new private RAID groups only (and not expand on existing ones). The only way to realign to consistent/recommended sizing is to redo the pool from scratch which means LUN migrations assuming you even have the space.

The following post talks about the logic behind the private RAID groups and provides a command to show how they were constructed.

https://community.emc.com/message/696115#696115

3) Pool 3 has a private RAID group that has a large deviation from its peer (to be honest, most of the pools have PRG's that deviate from the others in the pool but only noting the 2 with the largest deviations)

a) 4+4

b) 2+2

4) LUNS 91 and 92 together account for a large amount of the IOPs (they are practically all write)

- Both owned by SP B (candidates for separating LUN ownership?)

- Also accounts for higher overall throughput on SP B compared to SP A

5) Forced flushing, when it occurs, is mostly on SP B

- LUNs 117 and 116 are registering the most activity related to this

- LUN 117 is trespassed (current owner = SP B/default = SP A)

- When it occurs it of course affects all LUNs and will lead to high response times

- Review emc186107 for more information

6) Undersized  RAID Groups

- Focus most attention on RAID Group 27 (period between 10am and 12pm).  Then 24 and then 25 which are all undersized for the load being presented

- Review the individual disk IOPs and MB's that registered in excess of the documented rot: IOPs for 15K SAS drives = 180 (small block random) or 12MB/s (large block sequential)

- When write IOPs are higher than the documented saturation points, write cache can be heavily utilized

- You are underestimating the requirements for the LUNs currently on those RG's (and pools)

7) High queuelengths on LUNs (anything over 10 - 12 suggests it may have a noticeable impact)

a) 99, 66, 72, 74, 51 are hovering over this consistently

b) Others have registered spikes over 10-12

8) Overall SPB3 is (relatively speaking) underutilized (no queue full documented on any of the ports)

a) You mention all ESX 4, what is the PSP you are utilizing?  Default is Fixed with Array Preference which shouldn't be used (notably it was removed from ESXi v5)

b) I can tell you aren't using PowerPath as you'd generally see an even distribution across the front-end ports

9) Are your Mirrors fractured?

No activity during this period had registered on ports SPA 0 or SPB 0 which are of course the MirrorView ports

10) LUN response times >20ms

Should look into those periods for any that exceed this *generally* speaking, undesirable threshold

MISCELLANEOUS

==============

1) Check for trespassed user LUNs (notably pool luns due to its immutable allocation owner which if trespassed means it will now utilize resources from both SP's)

naviseccli -h getlun -trespass

Command above reports on both pool and FLARE LUNs (as if you clicked on "Trespassed LUNs report" within Unisphere)