Isilon Historical Storage Utilization

isi statistics hist -F --stats=ifs.percent.used,ifs.bytes.used,ifs.bytes.total

Check out:

isi statistics hist --help

for more controls (such as format, time range), and:

isi statistics list stats

for more items to query (lists all items for "live" queries, but only few of them are kept in the history).

-- Peter

Peter,

on my cluster hist seems to go back about 1 year, is that a default parameter ..can it be changed ?

Hi Dynamox,

It seems tempting to mess around with

/ifs/.ifsvar/etc/mcp/sys/stats_engine.xml

but better do that ONLY with support from support!

With SmartQuotas one has more choices for collecting and archiving:

isilon-1# isi quota reporting set --help

isi quota reporting set: Command help

Change quota config settings

'isi quota reporting set' options are:

  --report-dir=      directory to store scheduled reports

  --report-schedule=    schedule for generating scheduled reports

  --report-retain=          the number of scheduled reports to retain

  --on-demand-dir=   directory to store on-demand reports

  --on-demand-retain=       the number of on-demand reports to retain

  --help, -h                       Print usage help and exit

And, of course, crontab can be your friend.

(E.g., we log a full dump of isi client statisics every 10 minutes;

IT's humble contribution to piling up semi-structured "Big" Data...)

-- Peter

I appreciate the info so far, but I'm looking for something graphical. It is rather surprising that InsightIQ doesn't have a built in reporting mechanism that shows historical storage utilization. Any other thoughts? Am I missing something?

Do you see historical data for any values but for storage consumption;

or do you see no storage reports at all?

BTW: Beware of charts! While a storage growth rate of +60% p.a. is

already considered as "exploding", in a one year chart it

visually appears "linear".

Which can be highly misleading when deriving trends.

Always compute the past growth rate in percent or as a factor (like x1.6)

to make predictions. With +40% p.a one should be happy, with +100% p.a

one should expect trouble... (+60% p.a. is just what the progress of

technology has been sustaining in the long past).

-- Peter

Within the Isilon GUI, I do not see any storage reports at all besides the tiny little charts on the "Cluster Overview" page. In InsightIQ I don't see any historical disk utilization graphs either. InsightIQ has Data Usage reports, but they only show what data is being utilized at this moment and not historical.

Am I missing it here or did Isilon completely snooze this very basic feature off?

Have you checked the data resolution policies (under Configuration)?

The InsightIQ 2.5 users guide isn't terribly verbose on what is tracked,

but the list of exportable items on pp 54 might serve as a reference.

Not sure wether the stats I mentioned before will be

imported into InsightIQ retro-actively.

In addition to the lightweighted tracking there is full "File System Analytics",

which needs to be enabled and scheduled on the cluster itself(!).

Here, definitely no historical data exists from before the point the first run...

-- Peter

I'll look into the resolution policies. I'm not familiar with that.

The "FSAnalyze" job is running. It was my understanding that this job was necessary for InsightIQ to gather any statistics.

FSAnalyze is only for the file system analytics.  Even if FSA fails to run, the IIQ will continue to collects other cluster statistics.

Andrew - are you saying there is another job that runs & collects historical disk utilization?

Ok, so moral of the story is Isilon has no historical disk utilization features built into it unless I want to manually issue a bunch of CLI commands on a daily basis & export the results into an Excel spreadsheet.

Seems the FSA job isn't really working,

or InsightIQ doesn't get data from FSA,

or InsightIQ is configured (or restricted by space) to keep data only a for short time,

or you're expecting too much, like seeing data from before the point when FSA was initially set up.

Other than the built-in statistics, which exist mainly for troubleshooting,

and the freedom of a full UNIX system combined with live/scriptable isi statistics,

yes, there are only licensed tools at extra fees:

- SmartQuotas, which come with their own reporting (to XML, simple scheduling interface included)

- InsightIQ&FSA, which should give lots of data but might have pitfalls in setting up; really do check with support.

Good luck

-- Peter