PAPI API: How to get size of folder (recursively)

That information does not exist in the directory metadata so I would not expect to find it via the RAN API.  InsightIQ reports on directory sizes based on the FSA job.  Bottom line, you have to walk the tree to get the size.

While you could treewalk via the RAN API, I don't think you would want to. Based on my testing, it is much slower than NFS or SMB access.

It would be a very nice feature to add to the metadata. This would probably also make the FSA job much faster. FSA has been running for 10.3w in our system. We have a huge number of small files and new files are constantly added/removed. This seems to kill the FSA job, hence, we have not had one successful run since we installed the cluster.

Yes, we have both SSD og metadata accelleration activated. We only have 6TB disks. We have a mixed setup of s200 and nl400 nodes. Currently we have around 650m files. These are constantly renewed. I do see your point regarding the metadata. The scan takes around 1 week and the remaining time is the index build.

I've got around 1.5B objects in my mixed S/NL cluster with a lot of churn and the metadata on SSDs (except the 6TB NL410s, but that's another story). My fsanalyze jobs typically run phase 1 in about a day, then sit on phase 2 for a couple of days. Which version of OneFS are you on? And how many nodes total?

Thanks, but your config isn't very clear to me. With 6TB drives AFAIK the only acceleration mechanism is the L3 metadata cache, and it seems pretty tough to confirm at a certain point in time wether it got fully warmed up without overflowing. Need to watch the L3 metadata start/hit/miss rates continuously.

Cheers

-- Peter

With 1.5B objects, it's a pretty wide range of static vs changing, but there is a fairly high rate of change. You should have plenty of CPU (within the range of crappy CPUs Isilon uses) to handle this, so you're probably running into the whole phase 2 only runs on one core of one node problem. I don't know if that has ever really been resolved.

I am running 18 x S200, 8 x NL400. OneFS is 7.2.0.2. Are your files static or are they changing? I think I will give support a call regarding this. They did look at the job earlier, but that never fixed the issue.

This is a great discussion.  Thanks everyone for chiming in.

To answer the original question asked by spengler, the OneFS API (through RAN) does not currently provide that feature.  There are other ways to get at that information.  InsightIQ can, but it is dependent on the FSA job completing. If the FSA job isn't completing in a timely manner, InsightIQ would not be able to use that up-to-date info.

Isilon Product Management and Engineering teams have been focused heavily in this area.  Definitely stay tuned for announcements to the improvements in this area.  (This is all I am allowed to say on a public forum).

If your FSA job is completing, you could use InsightIQ's graphical interface to get at the directory size.  Or alternatively, you could use the iiq_data_export utility on the IIQ instance to export that information in an automated fashion (use case is if you are trying to do so with many directories).

See this thread -- Exporting InsightIQ data -- search for the string "iiq_data_export".

Hope this helps.

I know the job engine should have been re-writen on the Next release. Looking forward to this. As you can see, there are some strange results in our platform. The jobs seem to be running forever, but they just did complete 3 days ago (For the first time) since we got the cluster over a year ago.

FSAnalyze

Elapsed Time

1.67 d

Phase

1 of 2

Progress

Processed 313509682 LINs; 0 errors

LIN Estimate based on LIN count of 10 done on Dec 6 01:45:15 2015

LIN Based Estimate: - 40h 8m 21s Remaining (-1159870476% Complete)

MultiScan

Elapsed Time

3.01 d

Phase

1 of 4

Progress

Collect: 233985540 LINs, 0 errors

AutoBalance: 233985540 LINs, 0 errors

LIN Estimate based on LIN count of 10 done on Dec 6 01:45:15 2015

LIN Based Estimate: - 72h 12m 35s Remaining (-1955111896% Complete)

Block Based Estimate: 153h 1m 39s Remaining (32% Complete)

0 errors total

This sound interesting? Have you got any information on this issue?

My only guess is that the FSAnalyze didn't actually work, which threw it back with 10 LINs, thereby screwing up the percentages on the other estimates. But I dunno exactly what the problem is with your FSA.

> LIN Estimate based on LIN count of 10 done on Dec 6 01:45:15 2015

That's bogus but the problem comes from not picking up the right number out of a previous job report (with not necessarily needs to be FSA); it's not that this FSA job has failed after 10 LINs...

-- Peter