Enabling L3 Cache and populating it

Ah, big maintenance windows must be nice...

Hi Peter,

i tried this with my test-system. it's a physical Cluster which is only in use by ~10 Persons.

I ran a treewalk and waited a few hours. Then i started a second treewalk (2times in parallel). A day after i started a third set of treewalks... the results are quite... shocking...

BUT: i have a prefetch miss Ratio of 0.0% in L1 and 0.6% in L2. are the treewalks maybe sequential, thus L1/L2 prefetching data from L3 and these prefetchs do not count into L3 stats?

@Carlilek:

I don't have such a big maintanance window... but 95% of users are gone over the Weekend, which leaves the Clusters pretty idle for some "extrawork"

Regards

--sluetze

My L3 experiments (on my primary prod cluster....) were a total failure when it came to metadata. My cache emptied so fast that the metadata just didn't hang around. YMMV and certainly I had/have other issues going on, even after reverting back to metadata r/w.

These are cumulative stats beginning from reboot

or execution of "isi_cache_stats -z" (=zero).

Better zero the stats to start a new recording interval,

or use plain "isi_cache_stats" for real-time stats.

-- Peter

Hi Peter,

i zeroed after enabling L3 Cache.

Carlilek: quite interesting. If i understood the Technologie korrekt, the Data / Metadata will fill up the SSDs until they are full. then the oldest blocks will be discarded.

Well let it put me this way:

On this Testcluster I have more SSD-Space than Data and Metadata combined. So at some Point every file smaller 127K should be in L3 Cache...

-- sluetze

Ok, so what exactly is "shocking" here.

You have exceptional high hit rates on L2 level for both data and metadata.

The L2 misses are below 1% -- and notice how the L2 "miss" block rates

are showing up nearly identically as the L3 read start, for data and metadata respectively.

This is exactly how a multilevel/tiered cache system should work.

Now on L3 level there are no hits at all, which is kind of sad and unexpected.

Is this what you called "shocking"? Keep in mind that the absolute rate

of attempted L3 reads (in blocks/s or MB/s) is still low, almost "noise",

compared to the actual system throughput as seen on L1 and L2 levels.

I'd say that for the moment you are still "lucky" with your L2 cache and

these tiny amounts of L2 misses are stuff that had no chance to get cached

on any level so far.

Once your L2 hit rates diminish and you see substantial cache misses here,

it's when it really gets interesting for L3 data and L3 metadata...

Cheers

-- Peter

Hi Peter,

the shocking part for me were the low percentages in t he L3-Cache. Cause of this was, that there is just too less traffic and data on the box. thus all the data/metadata resides in L1 and L2.

In the last days the L3 metadata hits raised to 35,6% - but looking  on the numbers i see 93384 misses and 51633 hits. So i have 62790 new read.starts and 51633 hitted. That makes a very good 82% for this new hits.

Thanks for your support and your feedback.

Regards

--sluetze

How is the performance compared to your old configuration?

Hi carlilek,

until now i only implemented l3 on our test-Cluster. ill provide feedback after implementing it on one of our productive Clusters.

Regards

-- sluetze

Hi Ken,

as promised some Feedback so here it is:

my calculation based on the experiences in my testenvironment was wrong. I calculated with a time of 19h (max) and 8h avg but it took ~60h on my first productive clusters (not the smallest but not the biggest).

Performance:

improved. it is hard to say in numbers but regarding read-requests and data we reduced the amount of read-requests which do not get answered by prefetch or cache by ~60%

Hitrates of data in L3 are ~60% still increasing

Hitrates of metadata in L3 (not warmed up!) are ~40% (increasing)

we have a setup where we measure the value of a file getting written and read afterwards which also showed improvements.

Reagards

-- sluetze

Interesting. How metadata heavy is your environment? Our L3 (metadata only) has been catastrophic.

we do not have real defined workflows, since we use the Isilon not for applications but as NAS for home and groupdrives (nearly) exclusively. So most of the "metadata" actions are users traversing the folders.

If I look on L1 Cache I can see 43 metadata blocks for every datablock.

Also for metadata operations the bottleneck in our environment would be the latency since most access comes from offsite.

Ah, completely different environment, then. Our ops are ~50-70% metadata, virtually all onsite, with low latency switching and 10GbE. Interactive stuff is horrific when it's a first access for metadata on the L3-metadata pool.

Hello sluetze

how has it been going in the meantime?

There is a semi-obvious issue about L3 cache warming one should think through... (came to my mind during holidays)

The L3 cache is *local* to each node, as we know. This has consequences:

With metadata being mirrored on at least three nodes, any naive attempt to warm up the metadata cache by a single treewalk will only cache one mirror for each affected LIN.

Which means that later-on the odds of a cache MISS are still 2/3, or 2 against 1. 

Unless of course there is one "preferred" copy of a LINs metadata to be always used. Having this one in the cache would be sufficient for all subsequent accesses from other nodes.

Seems like nitpicking but obviously the exact choice which metadata mirror to *read* will have tremendous impact on L2/L3 cache efficiency -- further thoughts and facts, anyone?

Cheers

-- Peter