Ian_Marcroft
1 Copper

fsa export unable to go deep enough into the paths or multiple paths

Hi

Purpose: Extracting data from the FSA report via command line, in order to obtain size of file paths with top level directories and much deeper than displayed on iiq web gui.

Command:

iiq_data_export fsa export --cluster clustername --data-module directories --filter-rule directory:data/ifs/resolve  -o 6688 -n g.csv  --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq


The data that I could withdraw was good, but I was unable to go deep enough into the paths or multiple paths. I tried so many different possibilities, but none of them was able to produce what I needed.

Question:

is there a way to change the log size (extend report size) in order to obtain more detailed information from the paths, so sort if a --max-depth  so the report shows more details as the paths are deeper than is reported?


More detail:

  • iiq_data_export fsa export --cluster clustername --data-module directories -o 6688 --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq/

-works also: Gives out 1000 paths but not per directory?

  • iiq_data_export fsa export --cluster clustername --data-module directories  -m 1000 -o 6688 --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq/

-works: gives out all top level folders in FSA

SO WITH THE ABOVE COMMAND I WAS UNABLE TO GO ANY FURTHER INTO THE PATHS THAN TOP LEVEL DIRECTORIES

iiq_data_export fsa export --cluster clustername --data-module directories --filter-rule directory:data/ifs/resolve  -o 6688 -n g.csv  --path /mnt/172.27.210.1/ifs/data/ifs/eng/insightiq

THIS WOULD ALSO GIVE ME JUST TOP LEVEL AND NOT ANY FURTHER

Summary:

unable to query directory & directories (data modules & breakouts) within the same query...

unable to go deep enough into the paths or multiple paths

Any assistance greatly appreciate, has any of this functionality changed in v4.0, currently running 3.2.2

Tags (1)
0 Kudos
3 Replies
Peter_Sero
4 Beryllium

Re: fsa export unable to go deep enough into the paths or multiple paths

The report depth is limited to what has been scanned

by the FSAnalyze job of the cluster, but the depth level be changed.

(Prior to running a new job!)

In the InsightIQ WebUI, go to Settings -> [your cluster] -> FSA Configuration:

Default: 5.

Adjust the depth CAREFULLY, increment by one and run FSAnalyze.

Beware, the database might grow EXPONENTIALLY with the max directory depth!

hth

-- Peter

Ian_Marcroft
1 Copper

Re: fsa export unable to go deep enough into the paths or multiple paths

I extended to 25 and the reports were coming back at 40-50GB!

Even with this I could query each path individually, with extended depth results.

However from a top level query it wouldn't go further than the top folders within that path.

0 Kudos

Re: fsa export unable to go deep enough into the paths or multiple paths

Hi Ian,

I'm having trouble visualizing what it is you are having issues with.  With the iiq_data_export command for the directories data-module, you are supposed to only get top-level directories unless you specify a filter-rule.  With the filter-rule applied, you will only see the directory specified in the filter-rule as the top level directory and no farther.  To query for multiple directories, you'd have to issue iiq_data_export on each of them (using the filter rule).  The path option specifies the target directory where the CSV file should be stored (and not to be confused with the path you are looking to analyze).

I don't think this has changed in version 4.0.

There's one thread on ECN that covers a little bit of what you are trying to do: Exporting InsightIQ data

I'm pasting the relevant chunk here:

The following has been tested on InsightIQ versions 3.2.0 and 3.2.1.


On the node that is running your InsightIQ instance:


$ iiq_data_export fsa list --reports <cluster>

  Available Reports for: <cluster> Time Zone: PDT

    ID  FSA Job Start          FSA Job End            Size

--------------------------------------------------------------------------------

    765  Sep 30 2015, 10:00 PM  Oct 01 2015, 11:28 PM  3.500G

--------------------------------------------------------------------------------

$ iiq_data_export fsa export --data-module directories --cluster <cluster> --report 765 --filter-rule directory:TEST/MY/products/xxxx/vxx

    Successfully exported data to: directories_<cluster>_765_1443800888.csv

$ cat directories_<cluster>_765_1443800888.csv

path[directory:/ifs/TEST/MY/products/xxxx/vxx/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1443675617

TEST/MY/products/xxxx/vxx/test,1,0,0,0,0,2048,0

Note: The directory specified in the filter rule "TEST/MY/products/xxxx/vxx" -- this directory exists on the cluster as "/ifs/TEST/MY/products/xxxx/vxx".

Hope this helps.

0 Kudos