Isilon storage performance issue

Have you seen this?  I'd recommend to have look:

Ask the Expert: Isilon Performance Analysis

Maybe a few things can be checked in advance (before tracking things down to disk level):

- double check that no background jobs are running and stealing CPU or IOPS

- with four clients, is the network traffic well balanced across the four Isilon nodes?

- are the actual NFS read/write sizes large enough for 128K? (server and client negotiate a match within their limits.)

- is the random access pattern really in effect?

- for 128K reads, one could also try the concurrency pattern...

-- Peter

Thanks Peter for your valuable response!

-No background job was taking CPU/IOPS

-Need to check (Is looking client connection on dashboard of Management UI is right way?We are using smartconnect (with basic licence, I think , it is round robin policy)

-we did mount NFS on compute node using rsize=131072,wsize=131072 (need to check whether negotiated value is also 128k)

-we did setting in management UI, can you suggest ways to check whether it is really in effect or not?

-Need to check it (will update you once it is done)

> -Need to check (Is looking client connection on dashboard of Management UI is right way?We are using smartconnect (with basic licence, I think , it is round robin policy)

The WebGUI is ok, but IMO to slow for live monitoring.

On the command line interface (CLI):

isi nfs clients ls

isi perfstat

Other useful views for live monitoring (just to start with...)

isi statistics system --nodes --top

isi statistics client --orderby=Ops --top

isi statistics heat --top

isi statistics pstat

(these are 6.5 cmds, on 7.0 the syntax might vary, or use isi_classic instead)

> -we did setting in management UI, can you suggest ways to check whether it is really in effect or not?

isi get "filename"

isi get -DD "filename"

the latter shows (in very verbose form, but not so easy to count

the number of disks used) the actual layout of the file on the cluster disks.

Usually "streaming" access files should spread onto more disks,

but on small (or fragmented?) clusters the difference between

streaming/random/concurrency might appear minimal.

isi set -l {concurrency|streaming|random} -r g retune "filename"

will actually change the layout if needed. (I trust this more

that the WebUI). Even if finished, it might take some

more seconds until changes show up with isi get -DD

And if you have SmartPools enabled, make sure

you allow settings per file instead of SmartPools

ruling everything. (Or do use SmartPools, but then

you would need to run a SmartPools job each time to

implement changes.)

The access pattern (as listed by isi get) also affects

prefetching; try out all three choices. You are doing  random

IO, but on 128K chunks -- certainly larger than 4k or 8k.

And you have many concurrent accesses, so it's

hard to predict. It is even possible to fine-tune

prefetching beyond the presets for the three patterns,

but I would keep that for later...

-- Peter

Thanks Peter,

I will check the details and update you.(currently setup is not with me)

meantime I have some question to understand performance bottleneck , It will be nice if I get answers of below questions:

Considering I have four X200 nodes , and filesystem bock size is 8K and data protection is +2:1(2 disk or 1 node failure)

1.If I write file of 768 KB(128X6) size, will it write 6 data stripe and 2 parity stripe(128K size of each stripe and form a protection group of 8 stripe) on 4 nodes in a way where each node will have 2 stripes (at least one data stripe on each node ).

2.When it writes 128K stripe unit (16 block)  on given node , whereas node has 12 disk in my case , how many disk will it use? Will it use all 12 disk or write 128K on one disk only and for next stripe unit at this node use second disk and so on...

3.What will happen if my file size is exactly 128K , will it write  128K stripe on 3 nodes(mirror to ensure 2 disk or 1 node failure) and create overhead of 2x?  

4.If my application open a file and write on 128K chunk on NFS protocol and appending multiple 128K chunks to the same file , does OneFS use stripping for every 128K chunk or wait for multiple 128K chunk and do file striping latter to define file layout properly?

5.If I change filesystem block size from 8K to 32K , does it means my stripe unit will be now of 16X32K

6.OneFS uses 16 contiguous block to create one stripe unit , can we change 16 to some other value?

7.Can we access Isilon storage cluster from compute node (install RHEL) using SMB protocol, as I read in performance benchmark from storage council that SMB performance is almost double compare to NFS in terms of IOPS? 

Thanks & Regards,

Prakash

A couple of thoughts and suggestions:

http://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf

is really worth reading to learn more about the filesystem layout. And questions similar to yours

have been discussed here recently

Isilon overhead for 174kb files

How many sizes does Isilon consume when writing 128kb file with N+2:1 and N+3:1

With all information, it is really fun to examine a file's disk/block layout as reported by isi get -DD "file".

Furthermore:

> 5.If I change filesystem block size from 8K to 32K , does it means my stripe unit will be now of 16X32K

I don't think you can do so - which exact setting are you referring to?

> 6.OneFS uses 16 contiguous block to create one stripe unit , can we change 16 to some other value?

Couldn't  imagine, but the access pattern parameter controls whether a larger or lower number of disks per node are being used (under the constraint of the chosen protection level).

> 7.Can we access Isilon storage cluster from compute node (install RHEL) using SMB protocol, as I read in performance benchmark from storage council that SMB performance is almost double compare to NFS in terms of IOPS?

In benchmarks SMB IOPS appear higher than NFS IOPS because the set of protocol operations is different, even  for identical workloads, not to mention different workloads used. You cannot compare the resulting values...

For your original test, you might max out with the disk IOPS (xfers),  but you could also get stuck at a certain rate of your "application's IOPS " while seeing few or no disk activity at all(!) -- because your data is mostly or entirely in the OneFS cache . Check the "disk IOPS" or xfers, including ave size per xfer, with

isi statistics drive -nall -t --long --orderby=OpsOut

and cache hit rates for data (level 1 & 2) with:

isi_cache_stats -v 2

In case of very effective caching the IOPS will NOT be limited by disk transfers (so all that filesystem block size reasoning doesn't apply).

Instead the limit is imposed by CPU usage, or network bandwidth, or by protocol (network + execution) latency even

if CPU or bandwidth < 100%.

In the latter case, doing more requests in parallel should be possible (it seems you are right on that track anyway with multiple jobs).

To check protocol latencies, use "isi statistics client" as before and add --long:

isi statistics client --orderby=Ops --top --long

This will show latency times as:  TimeMax    TimeMin    TimeAvg   (also useful for --orderby=... !)

Good luck!

Peter

Peter, that is a great explanation.   

Could you explain a little more on the commands. I mean what do we have to look for that says something is wrong, when we type in the commands.

isi statistics system --nodes --top

isi statistics client --orderby=Ops --top

isi statistics pstat

isi statistics drive -nall -t --long --orderby=OpsOut

isi statistics client --orderby=Ops --top --long

Thanks,

Damal

Peter, thank you very much for the explanation. I hope other people who visit this page will make corrections, if required

>  With --long there are InAvg and OutAvg, which denote the request ("block") sizes (NOT the average of In and Out rates!!).  Small request sizes often indicate suboptimal configs on the clients side.

When you say small, any particular value that we have to look for ? Any recommendations on what configurations has to be changed on client side ?

Hi Damal:

for NFS, it's basically the max and preferred read and write sizes

(512KB, 128KB, 512KB, 512KB resp on Isilon side). Just

make sure that clients do not limit requests to smaller sizes via the

NFS mount params rsize and wsize (as max possible sizes).

These might me set in the clients stab or auto mount map,

some systems might have other places to set global defaults.

Random small IOs will result in smaller request sizes, of course,

as they can't get coalesced on the client side.

SMB1 is limited to 64KB requests which always has been pretty bad,

while recent SMB allows for much larger requests. "Secure packet signing"

reduces the allowed request sizes though. For potential restrictions on the client

side, please refer to the Windows (or other clients OS, resp. Samba) docs.

Cheers

-- Peter

Peter, thank you for the responses. It's gives a better insight on handling issues.

Best Regards,

Yoga

Fantastic summary Peter.  Thank you very, very much.

Amir