this is roughly how I use these commands to investigate
performance issues in new situations:
> isi statistics pstat
Bottom half: Overall view to see what's going on.
Compare Network -- Filesystem -- Disk throughputs
to see wether they are consistent with what is expected
for the workflow as far as it is known. In a typical NAS workflow,
network throughput should match filesystem througput.
Write throughput should match disk write throughput (mind the protection overhead).
Read throughput matches disk read throughput for uncached workflows,
or can be much higher which good caching.
High disk activity without network or filesystem activity indicates
some internal job is running (restripes etc).
High CPU without any network/filesystem/disk
activity would be very strange, for example
(some process running wild, etc)
Just illustrating how things can be learned from pstat.
Top half of pstat is protocol specific,
might need to run separately for NFS and SMB.
Again, quick check for consistency:
Does the observed mix of reads/writes/ etc. make sense
in the light of the assumed workflow?
> isi statistics system --nodes --top
Quick check wether the load is well balanced across the cluster,
and how the protocols are used (SMB vs NFS etc).
Can also indicate where physical network bandwidth hits the max.
> isi statistics client --orderby=Ops --top
Who is causing the load?
--orderby=In or Out or TimeAvg for throughputs or latencies resp.
> isi statistics client --orderby=Ops --top --long
With --long there are InAvg and OutAvg, which denote the request ("block") sizes
(NOT the average of In and Out rates!!). Small request sizes
often indicate suboptimal configs on the clients side.
> isi statistics drive -nall -t --long --orderby=OpsOut
Do the disk activities max out? Also --orderby=OpsIn or TimeInQ
Do the disk activities match the assumed workload:
Small SizeIn and SizeOut request size indicate metadata ops
or small random IO ops; pure streaming reads/write are usually upto 64k.
Disclaimer: Those are just by personal favorites (plus isi statistics heat),
and I might err on interpretations; I am not aiming at convincing anyone