How's your Fast Cache performing?

without knowing the details, it's likely the FAST cache was dumped to disk (cleared) during the upgrade and it will take time to promote data back into the FAST cache - could take hours or longer - that's the warm up period.

glewn

Thanks for the reply. Yes I'm aware of the "warm up" period. The data I took was about a week and half post upgrade so that should be plenty of time. I've been in contact with another customer with a much larger VNX implementation who has experienced the same problem on a larger scale. I'm hoping that EMC will acknowledge the change and at least explain what was behind making it.

Howdy,

I started a thread on the linkedin forums a while back regarding Flare 32 stability and us upgrading to that version. I had heard from EMC contacts that we should wait for several months due to some unknown patches coming out. We were anxious get to the mixed RAID group pools since 32 had been in the wild for 6 months so what was the problem I was thinking.

That thread generated quite a bit of discussion.. one key point was how a customer had seen a large drop off in Fast Cache hits especially for Writes. I then took some data prior to the upgrade and post.  I graphed the total read/write I/O vs cache hits for both datasets and the drop in hits is so obvious it hits you smack in the forehead.  That said our overall latency figures changed little of any. That may be because we're not pushing the array too hard just yet. I'm not a storage expert but logic to me says that if you're fast cache hits drop off this dramatically, more I/O is hitting spinning disk..

Thus begs the question why did EMC make this change and how should it affect me?

Hi Ron, we're about to shift our ESX data stores over from an older CX4 to a VNX5500 so our workloads are similar; just all kinds of random vmware guests doing different things.  I'm curious what kind of hit ratios you saw before and after?  More importantly, did you notice issues on the ESX/guest side (either in the data or noticed at the guest level) after the update that caused you to start looking at the statistics?

We're going down in spindle count slightly from our CX4 and were hoping between fast cache and the faster bus speeds, newer drives, etc., that we would not see a decrease in performance so your issue has peaked my interest; we're running the same flare that you updated to.

In R32 they changed the way the cache works for certain types of IO. This change is a good thing and should improve FAST cache performance by reducing the sequential IO from being promoted to cache and then never used.

In 31 and prior sequential small blocks were ending up in FAST Cache and were a huge part of the higher demand on it. We don't want to go to FAST Cache for such IO as SP Cache can deal with it much better and destage to disks much more efficiently

This information is found in the Release notes for FAST Cache for Version 05.32.000.5.006 – this is the ONLY release notes for FAST Cache for R32.

https://support.emc.com/docu42103_FAST_Cache_for_VNX_OE_for_Block_Release_Notes_Version_05.32.000.5.006.pdf?language=en_US

On page 3 you can find this change from R31:

Avoidance of sequential small block I/O

This capability reduces performance degradation caused by small block sequential I/O.

In the latest version of the FAST cache White Paper - White Paper EMC FAST Cache — A Detailed Review.pdf (Aug 2012) on page 11

https://support.emc.com/docu32136_White_Paper:_VNX_FAST_Cache_-_A_Detailed_Review.pdf?language=en_US

This is also detailed in Support Solution emc251589 and emc292696 (master list for FAST cache solutions).

glen

Unfortunately I started out with R32 so I don't have a basis of comparison but I can at least confirm that large sequential writes don't appear to be affected by having Fast Cache on or off, and I'm getting a zero cache utilization when doing them with benchmarking tools and nearly the same write speed either way; so this change does at least appear to be not promoting that kind of data into the cache, as it should.  What I am seeing though is an unexplained 40% drop off in read performance when cache is enabled for long sequential reads:

Seeing unusual benchmark results w/Fast Cache and Fast VP

You might check out this thread Is Flare 32 stable? | LinkedIn

One of the people who chimed in has experienced quite a bit of pain.

Thanks for the detailed reply glen. We must have a good % of sequential I/O in our profile that is now bypassing FC which as you say should be a good thing. It was startling to see the drop in hits thus begged the question that FC wasn't working as well as in 31. I'll try to see if we have higher SP Cache utilization in 32.

Rainer,

What kinds of metrics are telling you that it works better? I'm still learning how all this magic works and would appreciate any advice on how to better use things like Analyzer to assess things.

Ron

Hi,

there were changes in FAST VP as well and I wouldn’t rule out that maybe the way the hit rate is being calculated.

All of our internal testing shows that R32 FAST Cache works better than R31.

Rainer

Have you considered that fact that in your config you are reading from just two flash disks in case of FAST Cache vs. lots of spinning disks in your case of FAST Cache off ?

Flash drives aren’t that magnitude faster on sequential read – a sizeable number of idle magnetic disks can outperform two flash disk (with a specific workload).

In real life on systems with a production workload it isn’t always as simple than a benchmark on an idle system.

Throw in some other I/O and all of a sudden the disks are busier and your nice sequential I/O isn’t so sequential any longer.

Besides – Bonni++ might be convenient but it’s not a good benchmark considering locality.

Try using something like iozone – or better something that comes close to your application behavior

Or take your other question for example – writes being faster than reads.

At first glance we are used to that – writes are expensive.

BUT on the VNX in a not saturated state all we need to do for a write is to put it into write cache and mirror it to the other SP’s write cache.

That we do very efficiently – we don’t have to wait for a magnetic disk to be available, then position its r/w head, wait till we get to the data, …

But then what happens if the write cache gets full faster than we can flush ?

I/Os have to wait for a forced flush.

That is one example where FAST Cache helps since flushing to flash drives is faster than flushing to magnetic disks.

That also explains why in some benchmarks writing to blocks on FAST Cache isn’t faster than writing to blocks on disks.

Because you haven’t benchmarked how fast you write to disks – it was how fast you could write into a DRAM based write cache.

Not from metrics in Analyzer alone.

Storage side metrics are good but the real measure is what the clients are seeing in terms of IOPS, response time or bandwidth.

We are running a large number of benchmarks for each release at EMC.

Both artificial ones like iometer as well as internal tools and real applications like Exchange, SQL server, Oracle ….

Just talk to your local EMC performance guy (called Speed guru) and he can show you some of the improvements.

Rainer

Cpmplexity is the savior of IT

glen

Large (over 1MB I believe) sequential Writes also bypass FAST Cache. Large sequential IO's are not a good fit for FAST cache - you should avoid enabling FAST cache for those types of operations. In some cases, because of the size of the IO that is promoted (multiple accesses to the same 64KB chunk) if the sequential IO is 128KB and larger, it most likely will not get promoted.

See emc251589 for more Best Practive tips and see the attached White Paper - especially page 20

glen

VMware workloads would seem to me to obfuscate the I/O profile a bit compared to a DB server with drives assigned to specific LUN's of its own. To that end I've always struggled a bit when it comes to simplicity vs performance and complexity. Default settings throw all your vmdk's on a single LUN. You can go through the trouble of creating specific LUN's for logs and tempdb on non-fast cache enabled LUN's.. yet the first time the VM gets storage vmotioned everything is back on the same LUN. Not to mention you have to manage gobs of small LUN's on the array.