Ah, big maintenance windows must be nice...

Hi Peter,

i tried this with my test-system. it's a physical Cluster which is only in use by ~10 Persons.

I ran a treewalk and waited a few hours. Then i started a second treewalk (2times in parallel). A day after i started a third set of treewalks... the results are quite... shocking...

BUT: i have a prefetch miss Ratio of 0.0% in L1 and 0.6% in L2. are the treewalks maybe sequential, thus L1/L2 prefetching data from L3 and these prefetchs do not count into L3 stats?

@Carlilek:

I don't have such a big maintanance window... but 95% of users are gone over the Weekend, which leaves the Clusters pretty idle for some "extrawork"

Regards

--sluetze

My L3 experiments (on my primary prod cluster....) were a total failure when it came to metadata. My cache emptied so fast that the metadata just didn't hang around. YMMV and certainly I had/have other issues going on, even after reverting back to metadata r/w.

These are cumulative stats beginning from reboot

or execution of "isi_cache_stats -z" (=zero).

Better zero the stats to start a new recording interval,

or use plain "isi_cache_stats" for real-time stats.

-- Peter

Hi Peter,

i zeroed after enabling L3 Cache.

Carlilek: quite interesting. If i understood the Technologie korrekt, the Data / Metadata will fill up the SSDs until they are full. then the oldest blocks will be discarded.

Well let it put me this way:

On this Testcluster I have more SSD-Space than Data and Metadata combined. So at some Point every file smaller 127K should be in L3 Cache...

-- sluetze

Ok, so what exactly is "shocking" here.

You have exceptional high hit rates on L2 level for both data and metadata.

The L2 misses are below 1% -- and notice how the L2 "miss" block rates

are showing up nearly identically as the L3 read start, for data and metadata respectively.

This is exactly how a multilevel/tiered cache system should work.

Now on L3 level there are no hits at all, which is kind of sad and unexpected.

Is this what you called "shocking"? Keep in mind that the absolute rate

of attempted L3 reads (in blocks/s or MB/s) is still low, almost "noise",

compared to the actual system throughput as seen on L1 and L2 levels.

I'd say that for the moment you are still "lucky" with your L2 cache and

these tiny amounts of L2 misses are stuff that had no chance to get cached

on any level so far.

Once your L2 hit rates diminish and you see substantial cache misses here,

it's when it really gets interesting for L3 data and L3 metadata...

Cheers

-- Peter

Hi Peter,

the shocking part for me were the low percentages in t he L3-Cache. Cause of this was, that there is just too less traffic and data on the box. thus all the data/metadata resides in L1 and L2.

In the last days the L3 metadata hits raised to 35,6% - but looking  on the numbers i see 93384 misses and 51633 hits. So i have 62790 new read.starts and 51633 hitted. That makes a very good 82% for this new hits.

Thanks for your support and your feedback.

Regards

--sluetze

How is the performance compared to your old configuration?

Hi carlilek,

until now i only implemented l3 on our test-Cluster. ill provide feedback after implementing it on one of our productive Clusters.

Regards

-- sluetze