I am monitoring the performance of our large (56TB) metaLUN using Analyzer.
The box to which this is presented is experiencing large disc queues and the queues also seem high on the metaLUN - peaking at 27 and averaging about 6. The strange thing is that if I watch over a period I see the queues drop to 0 every 15 minutes or so, before rising straight back up to the previous value.
My initial thoughts are that this is being caused by cache flushing and that I may need to adjust my high and/or low watermarks.
If your host is experiencing a high number of queue busy 'errors' you should consider looking at your HBA queue depth - at least thats what it's called on an Emulex lp type HBA. - you obviously have a lot of physical spindles in your metalun, so the default number is not enough. On the CX3-80, look at the other numbers as well. If your average response time doesn't increase then it's not really maxing out your disks, also look at the queue busy parameter. If your cache is experiencing forced flushed then your disks are busy.
What is the layout of your metaLUN? Has the layout of the metaLUN changed recently? Concatenated or striped, and the number of spindles behind the components? Have you always seen this behavior with the queues?
I would check with EMC on this, but if you are using a striped setup it is possible that you could see delay in reading or writing a stripe with the large amount of spindles required to achieve a 56TB metaLUN.
Thanks for the replies. A little more information:
The metaLUN comprises 135 spindles 'under' 9 LUNs concatenated to produce the 56TB; this configuration has not recently changed.
I have only been monitoring the queues since our operations team noticed performance issues on the host to which the metaLUN is presented.
Some additional info: changing our watermarks from 80/60 to 60/40 in an attempt to stop forced flushes has not affected my 'live' queue graph which still looks something like this (excuse the ASCII art, the values are not constant I just want to illustrate the drops): _______ _______ _______ _______ _______ ............\/...........\/...........\/............\/...........\/..........
The time period between the drops (to zero) is exactly 16 minutes.
What application/OS is accessing the LUN? Is it random or sequential? Are the component LUNs in RAID groups by themselves, or are there other LUNs shareing the RAID Groups? You could just have a problem with one of the components, maybe sharing spindles with a busy application.
Individual LUNs are bound to individual RAID (5) groups (that comprise all 15 discs in a DAE). These LUNs are concatenated to form the metaLUN.
The host is running Win Server 2003, the application is bespoke, the reads and writes are large and sequential - media files averaging 5MB in size.
If I look at the components of the metaLUN in Analyzer there is really only one doing anything, that is the one currently writing new data. The 'older' LUNs (those that made up the original, smaller metaLUN - we have expanded it) are hardly used, except for the odd read (again sequential).
This is all one NTFS volume? How many files are on there?
We ran into a problem with a 2TB volume that had over 40 million files on it. As it turned out, Windows was having a lot of trouble dealing with our MFT. It was fragmented and managing the MFT is a single threaded process. There a few thing you can try:
Those solutions allowed us to get by for a short time, but eventually we had to change our approach. Our data was being created on a monthly basis, so we had folders for each year, and under each year there were folders for the 12 months. What we did was create physical volume for each month and mount it on each of the folders. This solved the problem.
Using your LUN size and your mean file size you probably have five to ten million files in this system. I have certainly seen Windows performance suffer under similar circumstances.
I have one file server where there are over 10 million files which "churn" slowly. The interactive response times are fairly reasonable, but bulk operations, like backups, become significantly longer as time goes on. The volume will not complete a defrag, but running defrag for a while (several days) periodically (every few months) will provide an improvment of several hours for the full backup runs.
perklit
6 Posts
462
1
Posted July 13th, 2007 01:00
On the CX3-80, look at the other numbers as well. If your average response time doesn't increase then it's not really maxing out your disks, also look at the queue busy parameter. If your cache is experiencing forced flushed then your disks are busy.