High Disk write latencies

It really looks like SPA is getting hammered.  are the lun's split evenly across both SP's?  If they are,  it might come down to the type of luns on each SP.  Maybe you have to many hi I/O R1/0 luns on SPA or something.

I got this from the Analyzer Guide:

The higher the queue length for the SP, the more
requests are waiting in its queue, thus increasing the
average response time of a single request. For a given
workload, queue length and response time are directly
proportional.

Looks like your response time and Queue length are hi which would make your Utilization high also.

Check the logs for any background verifies also as they will place a load on the SP.

Hi,

You are right I feel SPA is being thrashed so I am checking the highest utilised luns and trespassing them to SPB, but it seems to be a painfully slow process since I cant find a way to easily identify the highest utilised luns from analyser.

By response times did you mean the reponse times shown by both SPs are a cause for concern? If not then, since I trespassed the problamatic luns to SPB should the SPA response time still affect it?

Also is there a base value above which we say the response time is bad?

Thanks

one way to find the highest util luns on a SP is to right click the SP>Performance Detail> and just check all the lun's.  the utilization box is checked by default.  this will give you an idea as to which ones are the highest.

No i was saying that the SPA response times are high comparded to SPB.  moving that lun from SPA to SPB should have made the response time on B go up and A go down.

I havent found a document stating what  are "ok" values for any of the polled events so i cant be sure.

The SP response times and queue lengths look fine to me, generally less than 10 on average is what I like to see on an array. This is a number that I have also seen referenced by EMC personnel but I cannot find the reference at the moment.

As kenn2347 said you can use analyzer to spot your busiest luns. Use the Peformance Detail view and select all the LUNs in the array: on the LUNs tab right click on LUNs > Select All > LUNs. Then look at the Total Throughput (I/Os), Queue Length, and Response Time metrics. You will have to look at the metrics individually as they will have different scales and it will be hard to tell what the highest performers are if all the metrics are viewed together.

Thanks for your replies. Relieved to know SP response is not much the issue here. Yeah, I did follow the same method to identify and trespass the luns, since I have what Emc call a large configuration the navi analyser window takes a bit longer to load and come up with the details and then going through the same is also a bit slower, am thinking of looking up on analyser cli to find something useful then probably I can script it.

Looking at the lun and RG values does anything seem out of place which could lead to these disk latency errors seen on the windows server hosting exchange. Is there something I am overlooking and which could give me a better understanding of these latencies?

Reg,

Ash

The 5 disks which make up the RG ========================= service Time(ms)      = 3.3 to 4.5 Avg busy queue length = 4.5 Throughput (IO/sec)   ,Avg=595839 /Max=10709894

595k IOPS combined for 5 drives ? (5 x 180 IOPS = 900 IOPS)

Thanks Ash, those numbers make sense.

So, looking at the numbers though it looks like you shouldn't have any peformance issues with the LUN. A single 15k FC disk can handle sustained 180iops and bursts even higher. So the fact that your host is reporting a latency of greater than 20ms is odd.

What exactly is being hosted on this LUN? Is it just one Exchange store? Multiple Stores or Stores and Logs?

   apologies for those grosly mis-sstated  numbers... please find the correct values below

RG
==
I/O/sec :    Max = 398.4 / Min = 30.5 / avg = 124
MB/s    :    Max = 5        / avg = 2.8

Per Disk
========
I/O/sec :    Max = 80    / Min = 6.3 / avg = 25
MB/s    :    Max = 1        / avg = .57

Lun
===
I/O/sec :    Max = 317   / Min = 8.7
MB/s    :    Max = .3    / avg = 1.4

You are right the IO on this lun is minimal so I suspected the SP but the trespass did not help and yes the read and write cache are enabled for the lun.

I am exactly not aware of the exchange design and the number of stores on this lun, but I will try and get the host config and application details from the resp teams.

Thanks for all the feedbacks,

Ash

I assume you checked to make sure that read and write cache is enabled for the LUN?

Be sure to check the SP Event Log for both SPA and SPB - see if you see any excessive "trespass"  messages - excessive is more than a 20-30/minute.

glen