Caching in clusters without SmartPools license

The L3 cache is per node pool, not per file pool (aka SmartPools file policy).

Several stats are available, but it's a bit tricky to get the full picture,

or to predict the result.

You can check the metadata read latencies for your NAS protocols,

by client OR by specific operation:

isi statistics client --class namespace_read -t --orderby TimeAvg

isi statistics proto --class namespace_read -t --orderby TimeAvg

(Ops from SyncIQ, NDMP, AutoBalance and other jobs are not included here.)

Observed latencies can be very low already, due to effectice L1 and L2 caching,

and you might NOT even clearly see which ops come from the current SSD

acceleration.

But after migration to L3 cache, metadata/namespace ops that

miss the L1-L3 caches and thus come from HDD should clearly stand out.

In either case, you can look at the actual SSD IOPS and their characteristics:

isi stati drive -nall --type ssd --long

Some very simple insight into the caching itself is provided:

isi stati query -nall -snode.ifs.cache.oldest_page_age

(age in seconds)

With a larger cache (added L3) the caching intervals

will increase. It depends on your workload wether it

it will benefit from an increase that goes from, say

2 minutes (L1/L2 RAM) up to estimated 20 minutes (SSD L3

assumed e.g. 10 times larger that RAM). When streaming

many huge files, none recurring for hours,

20 minutes might be as useless as 2 minutes....

The total perspective vortex of caching stats is

isi_cache_stats [-v] [interval]

(or same stats, but through customized queries with:

isi statistics query -snode.ifs.cache.l1.meta.read.hit,... etc)

Run isi_cache_stats in interval mode (few seconds),

because the first output are the total stats since booted.

The -v verbose option helps to understand the

structure of the output, before going to default single

line mode. Note that the output is for the local node,

and that cache ops here include more than the NAS ops

(i.e. covering SyncIQ, NDMP, AutoBalance etc.)

The metadata read hits are often very high (80-100% rates)

already, but this comes from the fact that highest level directories

such as /ifs and /ifs/data are implicitly accessed so often.

Finally, a general consideration in addition to what Jim said:

If you have a classic many-small-files scenario and

are traversing huge hierarchies most of the time,

without revisiting the same spots frequently, then

you would need to rely on fast metadata everywhere

and always present. In other words, the "old" style SSD

acceleration really suits best for such cases.

Just got pointed to this brand new White Paper

File System Caching Infrastructure

in-depth explanation of L1/L2/L3 caching;

usage advice for SSD  L3 cache compared to traditional SSD metadata acceleration, including coexistence of both in one cluster;

and a very neat prediction procedure on p19!

Peter,

Great guidelines, thanks !

And that WP is good, little light on metadata caching details.

Follow up question :

How can I change SSD Strategy for a directory with cli, not webgui ?

isilon-1# isi set --help

usage: isi set [-fFLnvrR] [-p ] [-w ] [-c on|off] [-g ] [-e ] [-d <@r drives>] [-a default|streaming|random|custom{1..5}] [-l concurrency|streaming|random] [--diskpool ] [-A on|off] [-P on|off] [--strategy|-s avoid|metadata|metadata-write|data] file_or_lin ...

for example:

isi -set -v  s metadata myfile

isi -set -R -v  s metadata mydir  (-R for recursive subtree; will also apply to new files created here AFTERwards)

check with:

isi get -D myfile

isi get -d -D mydir   (single dir)

isi get -R -D mydir   (full subtree)

Folks,

Be very, very (very) careful when using isi set.  You are, in effect, taking the automatic management away and doing it by hand.  Down the road, this can and does lead to (how shall I put this) difficult/complicated situations, especially if SmartPools is invoked later on but the manually managed directories are excluded because an isi set was previously done.

Again, be careful.  For all, work with your teams to obtain an eval SP license if you need it.

Thanks again Peter.

One nice thing with SmartPools is that one can explicitly choose wether to keep or to overrule the manual per-file settings.

And one more nice thing is that in fact the eval license can be easily obtained.

Peter,

can you elaborate on "explicitly choose wether to keep or to overrule the manual per-file settings".

I did not find a way to specify that a filepolicy can override the "Manually Managed" protection settings. I'm still running 7.1.0, is this new in 7.1.1 ?

It's on the global SmartPools settings page, and applies also to FlexProtect where SP is not licensed.