If you do upgrade, make sure you export your data FIRST. For some reason, EMC/Isilon doesn't support just dropping the data into a new instance of IIQ post upgrade, so if something goes wrong, you'll be up a creek.

Also, 3.2.1 doesn't work quite right, so make sure you're on 3.2.2.

Thanks Katie, would you happen to know if those reports can be sent via email automatically? and or is it possible to automatically pull those reports for processing through a script?  doing it manually is a horrible.

I had to go to File System Reporting -> File System Analytics -> Select a report date -> Click on View Report -> then click on the Directory -> then Download CSV.

Imagine if you have 3 months worth of data.. and moving forward I dont want to go to insight everyday for download the CSV file.

i guess i will be staying on 3.1.1 unless 3.2.2 will allow me to get the data i wanted automatically..

Hi brutuz,

I'm a Technical Marketing Engineer, working on the Isilon product line.  I'm covering InsightIQ technicals in this new role.

I'm in the process of pulling together a list of blogs to answer some of these common questions.  Yours is a very common ask.  While I'd like to offer a more thoroughly documented process, I'll provide a slimmed down version here to show what this product can do today.

Since you intend on pulling the filesystem knowledge fairly frequently, tinkering with the GUI can be painful, as you stated.  I would try using the CLI.

The following has been tested on InsightIQ versions 3.2.0 and 3.2.1.

On the node that is running your InsightIQ instance:

$ iiq_data_export fsa list --reports

  Available Reports for: Time Zone: PDT

    ID  FSA Job Start          FSA Job End            Size

--------------------------------------------------------------------------------

    765  Sep 30 2015, 10:00 PM  Oct 01 2015, 11:28 PM  3.500G

--------------------------------------------------------------------------------

$ iiq_data_export fsa export --data-module directories --cluster --report 765 --filter-rule directory:TEST/MY/products/xxxx/vxx

    Successfully exported data to: directories_ _765_1443800888.csv

$ cat directories_ _765_1443800888.csv

path[directory:/ifs/TEST/MY/products/xxxx/vxx/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1443675617

TEST/MY/products/xxxx/vxx/test,1,0,0,0,0,2048,0

Note: The directory specified in the filter rule "TEST/MY/products/xxxx/vxx" -- this directory exists on the cluster as "/ifs/TEST/MY/products/xxxx/vxx".

Hope this helps.

Hi Rob,

     This is a great help! After playing around with it.. I was able to get all the information I needed..

DATEREF=$(date -d yesterday +"%b %d %Y")

REPORT_ID=$(iiq_data_export fsa list --report MYISILON | grep "${DATEREF}" | awk '{print $1}')

iiq_data_export fsa export --data-module directories --cluster isi-01 --report ${REPORT_ID} --filter-rule ${DIRECTORY}

RETVAL=$?

# Process

${WORKDIR}/process-iiq-report.pl

i noticed that iiq_data_export is a python code.. you can actually extend it.. though i need to learn python..

and also if there is documentation on the schema on the files stored in the /datastore files that would be great.. IIQ is using sqlite3 as db to store all information..

Thank you,

Good to hear!

In terms of the historical performance metrics repository that is InsightIQ's "/datastore" -- many customers have asked us about a published schema.  Unfortunately these do not exist.  Going forward, we urge customers to use the Isilon API (old name "Platform API") to query for performance metrics from the cluster itself, instead of trying to mine the datastore in InsightIQ.  Everything performance related that InsightIQ gets from the cluster CAN be obtained through Isilon API.

Also as of IIQ version 3.2 (code name Lumen), IIQ's datastore (which stores only historical performance metrics) has been changed to PostgreSQL -- making it leaner and faster.  So trying to mine IIQ data is only going to be short-lived.

Thanks again for your feedback!

I noticed some minor error:

path[directory:/ifs/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1444168813

nfs-shares,272737,7593935,0,54654,44072371405990,55915296110080,0

the logical size here is bigger than physical size but when it gets presented in IIQ web interface the physical because the bigger value..

i guess the the header in the result for the iiq_export is wrong...

Rob,

I "re-sell" storage to my internal customers who would like to be able to look at trending information in FSA to help with storage budgets.  Specifically tending information on quota utilization. My customers would like to login to IIQ, and look at their directory quota and see how storage utilization in that quota increased over time. Is quota related information exported from IIQ as well ?  I would be ok with throwing more storage at the IIQ appliance and have this information delivered by IIQ instead of my exporting it and rolling it into my own plotting tool.  Thoughts ?

hi dynamox,

  i started using IIQ version 3.0.1 and after i upgraded to 3.1.1.0022 to be exact.  They have included "Quota Reporting" under "File System Reporting" which as far as I remember it wasn't there.

  and the tool the rob provided can be helpful as well if you want to export tons of data with ease.

hth,