VMFS on SATA - Yes, I'm nuts...

play around with write cache ? In some instances disabling write cache on the LUNs might produced better results than having write cache enabled.

Ok here is an explanation of what you heard and what we normally doing with all ATA disks drive when the entire clariion exprience performance degradation .

Premise , clariion cache is global , so all the luns share the same cache , if a host write intensive to a lun and the destage of the data from the cache to the spindles is slow due to the type of the disks drive, like ATA drives you may fill up the entire clariion write cache and force the clariion to perform a force flush .
Force flush operation destage all the page in cache to the disks and pause the host i/o , as you can imaging this operation hurt the overall performance .

Force flush occur when we pass the high water mark .

The correct way to understand if you are in this situation is to retrieve a nar files and check the following :

- % dirty pages , if the % of dirty pages is most of the time above the high watermark this mean you force the clariion to perform a force flashing

- Cpu utilization need to be below 70%

- check for any trespass activity

there are 2 different approach we normally use to alleviate this problem :

- the first approach is to decrease the values of high and low watermark . For example if the high and low are set to 80 and 60 , you need to decrease those values to 60 and 40 .

- If you don't worry about performance on the ATA luns you can disable the write cache on the ATA luns .

I will recommend you to open a ticket with the clariion support and submit a set of spcollect with a nar files , there are million of reason why you can exprience a performance degradation . Consult one of the people in the clariion support before apply any change to your configuration .

Also if you fill up the cache probably you not use the right disks drive technology for the type IOPS you want to accomplish and will be better to evaluate to migrate the data to different drive type (SATA,Fiber)

Regards

like ATA drives you may fill up the entire clariion write cache and force the clariion to perform a force flush .

Interesting. So disabling the write cache on the SATA LUNs could sidestep the problem entirely?

I have SP collects and NAR files from this unit. Is there a tool or a method that I can use to analyze them?

Thanks.. We have not licensed analyzer. We have a month-long support case open with zero result, other than the replacement of a disk with a few soft errors. I don't know how to get them to stop asking for host grabs, logs, SP collects, and NAR's again and again and start actually analyzing performance for us.

If you have the license you can use navisphere analyzer and see the % of dirty cache you have but as I said before I will reccomend to involve the clariion support before arrive at any conclusion .
Regards
Pierluca

Please send me the service request number I will see what can I do for you .
I'm not in the support but I will try my best for help you .
I think if you not get an aswer from the support you shuld contact your sales rep or the local cs manager.

Thanks
Pierluca

This is interesting.

I'm using SQLIO running in a VM as a quick go/no-go test. Yes, I know it's not very accurate.

Turning off write-cache made put the SQLIO numbers in the toilet regardless of what the LUN was doing. Turning off the read-cache (with write-cache on) hurt IOP's by 40% when the LUN is quiet but it boosted the IOP's when heavy writes were occurring (via cloning a VM) by a factor of 5. Ok, this is somewhat better.

The truly impressive results happen when I turn ON all caching then turn OFF prefetch! I get IOPS of about 50% of our tier 1 FC LUN's when quiet and about 35% when busy. This is what I'd expect from cheap SATA!

After about five minutes of letting that VM clone operation run, it all falls apart. I think the cache high-watermark is being hit and a forced flush is occurring (as described above). Still, this is WAY better.

I'm amazed that our SP's only have 3GB of cache. Is that normal?

Yes. The Cx500 was the mid-level model of the CLARiiONs from two generations ago. The mid-level model of the next generation, the Cx3-40, had 8GB of cache.

Richard ,

Using sql io is not good for the type of the disks you have .
SATA /ATA disks are good for large block i/o single streaming applications , like backup .
SQL i/o is random 4kb .

For testing your ATA disks you need to try to run a single stream test write with at least 64K io size .

Regards
Pierluca

Thanks for your help. I didn't see a way to e-mail you directly. Our support case is 25955358. If you can find anything in the attached logs to explain the strange performance, we would be very appreciative.

Richard, I checked this case, 25955358, and the case is listed as closed (probably got closed by mistake when the CE that came out to install the Analyzer package closed his task).

For performance cases, support needs a number of logs in order to work the case, they need Analyzer files covering the time frame that the problem is occurring, they need the spcollects from the array to check on errors, they need host grabs in order to check the configuration of the servers, and some times they need the switch logs if they can not find any issues on the array or the hosts.

I looked at the case notes and see that the problem is primarily with the ATA disks in Raid Group 14 and 15. The older ATA disks are 5400 RPM ATA disks. These disks when used in a Raid Group must have all the LUN in the same Raid Group owned by the same SP - for example, if you have three LUNs in RG14 and all LUNs are currently owned by SPB. In RG 15, one LUN is owned by SPA (but currently trespassed to SPB) and the other is owned by SPB. So in this case the LUNs are correctly conifgured except that you need to make SPB the default owner for LUN 11 .

In addition, the original ATA disks were recommended for backup or low IO loads or sequential IO operations, no high throughput, random IO load. These disks can handle about 50 IOPS or7 MB/sec. We also recommend that they be used in Raid 3 rather than Raid 5. When used where they are receiving a lot of random, write IO, the performance on the array will suffer if these LUNs are pushing the Write cache to 99% Dirty Pages. You can check this on the array by right-clicking on the Array icon in Navisphere and selecting Properties. On the General page, ensure that Statistic logging is Enabled. Then select the Cache tab and at the bottom of the screen, you will see the % Dirty Pages - this will update once a minute - if you see it hitting 99%, then you have experiencing a Force Flushing issue. This is probably being caused by the ATA LUNs getting more Writes than the disks can handle. There are two solutions: disable Write cache on the LUNs or lower the Write IO's to the disks.

I will send you the Best Practices guide for Release 24 - this version still has information about the older ATA disks (not to be confused with the current 7200 RPM SATA disks currently shipping).

regards,

glen