Peter_Sero
4 Beryllium

Re: Ask the Expert: Isilon Performance Analysis

> OneFS's disk scale-out model does do a good job of leveraging all the disks in the system. If a drive has too many queued I/O's we will still queue to it. When you are in this state you should notice from the isi statistics drive -nall -long --orderby=timeinq | head -14 and then repeat into tail -14, that your top 10 and bottom 10 drives have a uniform degree of queued I/O.

That would be perfect, but what I actually see is

on 3-node X200 pool: TimeInQ  top 2.0 - 3.7 ms,  bottom 0.1 - 1.9 ms   => pretty uneven I think

on 3-node 108NL pool:  TimeInQ  top 4.1 - 4.7 ms, 3.0 - 3.3 ms         => ok, maybe quite even

As there are no parameters for adjusting the individual choice of disks

(other than striping etc.),

we simply accept OneFS's heuristic as it is;

you have explained  the rationale very clearly, thanks!

-- Peter

0 Kudos
Peter_Sero
4 Beryllium

Re: Ask the Expert: Isilon Performance Analysis

Thanks a lot for this clear overview of the caching concepts and mechanisms in OneFS.

The -z in " isi_cache_stats -z" probably exist in 7.0? 

With 6.5.5.11, no -z, and we still get different numbers with and without -v.

(Human readable, without -v,  still have "M" for Mega even in the

delta lines, just like in the posted example. Bug?)


I'm still struggling on how to make best use of node.ifs.cache.oldest_page_age.

It obviously reports the age of the oldest of any cache page in the

node, be it L1 or L2 and data or metadata. I assume no more

detailed montoring is available at this time.


-- Peter

0 Kudos
Peter_Sero
4 Beryllium

Re: Ask the Expert: Isilon Performance Analysis

Let me follow up on the caching topic with a question about SSD metadata acceleration.

With "Metadata on SSD", many users including us have found

a speed up of directory traversals by a factor of 10, which is very nice.

Actually this setting means: metadata for directories as well as for files,

and it can consume quite some amount of SSD space.

(Just as you mentioned there are large metadata structures for large files.)

Now,  Smartpools allows for rules to be applied to directories only.

This takes comparatively small space from SSDs, and therefore sounds promising

for sharing SSDs across pools (i.e. GNA - Global Namespace Accel.).

Would you expect a substantial speed up for directory walks by this?

What would be a good use case for applying a Smartpools rule to

directories, but not to the files inside?

(I have tried it out - but maybe got it wrong: at least no speedup

could be seen for a simple "find".)

-- Peter

0 Kudos
Haakon1
1 Copper

Re: Ask the Expert: Isilon Performance Analysis

Hello John,

I'm using Nexxus 7K switches with configured vlans and not seeing the performance we'd experienced when the system was installed.

Is there anything you could recommend to eek out a bit more performance?

0 Kudos
Haakon1
1 Copper

Re: Ask the Expert: Isilon Performance Analysis

How much of a performance hit would we experience if we enabled Global Namespace Acceleration - GNA and then fell below the 2% SSD capacity required?

0 Kudos
cassij
1 Nickel

Re: Re: Ask the Expert: Isilon Performance Analysis

On 6.5.5, we keep 1 copy of the metadata on the SSD and the other protection blocks for directory and inode on sata/sas disks. In addition to inode metadata we store other OneFS metadata blocks on the SSDs. In 7.x we introduced two metadata option METADATA-READ and METADATA-WRITE. The -READ is the same behavior as mentioned, 1 block on the SSD. The -WRITE option places all metadata copies on the SSD's. The goal of -WRITE is to optimize file creation, deletion, rename or namespace_write (setattr) operations.

SSD's should speed up directory tree walks were the inode/btree for the dir is on SSD. If you have two storage pools and you have a policy that as files age that you move them from nodepool with SSD to nodepool that has no SSD, it's possible that in this case that the FTW (file tree walk) is only traversing the sata/sas disks.

On NFS, when you walk a file tree the optimizer in most cases is to return more stat values as part of readdirplus. The default on NFS is 10 returned entries, Increasing this to 21 may help if your client is using readdirplus

isi statistics pstat --top, look at rdirplus operations. You will find the readdirplus option on the nfs export in the advanced settings.

As I mentioned in prior thread, the other actor in FTW is the kern.maxvnodes which increases with memory to 2,000,000 at the 32GB+ of memory. Being able to cache more vnodes will greatly effect performance. memory is faster than even SSDs.

isi get -D <dir/filename>| head -3

POLICY  W  LEVEL PERFORMANCE COAL  ENCODING      FILE              IADDRS

default  8+2/2 concurrency on    UTF-8         smb1000_32k.dd    <3,11,1911662080:512>, <4,11,1473441792:512>, <5,11,5808640:512> ct: 1370436556 rt: 0

In the above if you look at a dir or a file, the IADDRS, tell you were the inode lives. The <3,11...> Tells you that that IADDR is on LNN (logical node 3) and LDNUM 11.

isi_nodes %{name} %{id} %{lnn}

tells you the node ID and Logical Node number.

isi devices -a status -d 3  # this will list all the disk devices on node 3.

From this information you can now map which SSD's or SAS/SATA disks have inodes IADDR's. If metadata is set to be stored on SSD's you should find atleast 1 is mapped to 1 SSD or in the METADATA-WRITE case that all inodes are mapped to SSD's.

To finish this thread off you can see your OS cache of vnodes by looking at

isi_for_array -s sysctl vfs.numvnodes

b5-2-1: vfs.numvnodes: 863478

b5-2-2: vfs.numvnodes: 812164

b5-2-3: vfs.numvnodes: 885797

b5-2-4: vfs.numvnodes: 814139

b5-2-5: vfs.numvnodes: 890553

Best,

John

0 Kudos
Peter_Sero
4 Beryllium

Re: Ask the Expert: Isilon Performance Analysis

John, thanks for starting one more wonderful reply, but is there a technical or transmission problem?

Your post ends somewhat abruptly; and maybe also another post where you mentioned kern.maxvnodes seems to be missing...(?)

-- Peter

0 Kudos
SandiG
1 Nickel

Re: Ask the Expert: Isilon Performance Analysis

Hello John,

I had a colleague who had a performance issue that was mostly narrowed down to misaligned host partitions. After realigning the file system the performance was much improved, Can you help me understand why this makes such a significant difference?

0 Kudos
cassij
1 Nickel

Re: Re: Ask the Expert: Isilon Performance Analysis

What performance test(s) are you running and what result are you seeing ?

0 Kudos
cassij
1 Nickel

Re: Re: Ask the Expert: Isilon Performance Analysis

The typical reason a misaligned partition on a GuestOS creates a performance issue is that the blocks from Partition Table one as they are executed on a datastore straddle two stripe units of a protection group

With +2:1 protection on a 5 node cluster your N+M layout will be 8+2. 8 datablocks and 2 parity blocks

  [ 128KB ] - SU1 <---+ If your P1 partition starts in the middle of stripe unit 1 and you perform a 128KB I/O. This IO straddles

  [ 128KB ] - SU2 <---+ SU-1 and SU-2. Meaning you see 2 I/Os instead of one.

  [ 128KB ] - SU3

....

  [ 128KB ] -SU8

  [ 128KB ] - P1

  [ 128KB ] - P2

0 Kudos