Highlighted
2 Bronze

How can I reduce the size of isi_gather_info?

Jump to solution

Hi,

Our customer has a Isilon clustoer running in OneFS 8.0.0.3.

And the size of isi_gather_info has been too big, almost 3GB.

I'd like to know how to reduce the size of isi_gather_info.

I found that the "minidump.tar" file in a node was 2.5GB.

And I found a commnad option for isi_gather_info when Iread CLI administration document for OneFS 8.0.0.

--clean-cores

--clean-dumps

--clean-all

Which command delete the minidump file, and reduce the size of isi_gather_info???

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
3 Argentum

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Hi sa.taku,

if you don't have issues with your cluster and the DellEMC Support does not need a total log set with all dumps you can delete the dump files and core files with --clean-all or of course manually out of the /var/crash directory too. Memory dumps are created e.g. if there are failures or exceptions during booting a node. If you delete the files and after the next reboot there are new dump files you should open a SR to find the reason for the issue. Maybe there is something going wrong with the cluster...

If you want to keep all dumps and core files stored on the Isilon but want to exclude them of a new log set you can use the parameter --no-dumps and --no-cores.

Phil

View solution in original post

11 Replies
Highlighted
3 Argentum

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Hi sa.taku,

if you don't have issues with your cluster and the DellEMC Support does not need a total log set with all dumps you can delete the dump files and core files with --clean-all or of course manually out of the /var/crash directory too. Memory dumps are created e.g. if there are failures or exceptions during booting a node. If you delete the files and after the next reboot there are new dump files you should open a SR to find the reason for the issue. Maybe there is something going wrong with the cluster...

If you want to keep all dumps and core files stored on the Isilon but want to exclude them of a new log set you can use the parameter --no-dumps and --no-cores.

Phil

View solution in original post

Highlighted
3 Argentum

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Hello sa.taku,

To add to what has already been said, if you don't need a full log set. you can also run an incremental log set. Essentially it only gathers information that has changed since the last log set was run. Support may need the full log set, including cores and dumps, so it is best to check with the technical support engineer assigned to the service request as to what is needed.

The command to run an incremental gather is:

#isi_gather_info --incremental

The incremental flag is discussed on page 105 of the OneFS 8.0.0 CLI Admin Guide
https://support.emc.com/docu65065

0 Kudos
Highlighted
3 Argentum

Re: How can I reduce the size of isi_gather_info?

Jump to solution

You can also clean up the crap that Isilon doesn't and tech support doesn't want.  Things like old /ifs/.ifsvar/modules/jobengine/cp/ files.  There can be tens of thousands of these and the log extractor that the logs go through doesn't even extract them.  So we tar them up, compress them, send them across the Internet to EMC, where they throw them away...

0 Kudos
Highlighted
1 Copper

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Is it known to be safe to remove the old directories in /ifs/.ifsvar/modules/jobengine/cp?   A quick look shows that we have the whole sequence back to job No. 1.

On one cluster that is 14,000 directories and on another it is 25,000 files.  For the latter, the isi_gather_info takes 2-3 hours to complete, and if removing these made it faster I'd be very pleased.

0 Kudos
Highlighted
3 Argentum

Re: Re: How can I reduce the size of isi_gather_info?

Jump to solution

I regularly do my own cleanup.

#!/bin/bash

isi_for_array "find /var/crash/ -name 'hwcheck*' -mtime +30 -maxdepth 1 -type d -exec rm -Rf {} \;"

find /ifs/.ifsvar/modules/jobengine/cp/ -mtime +30 -maxdepth 1 -type d -exec rm -Rf {} \;

find /ifs/data/Isilon_Support/pkg/ -mtime +30 -maxdepth 1 -type f -exec rm -Rf {} \;

This has not caused me a problem yet on any of my clusters.

Highlighted
2 Bronze

Re: How can I reduce the size of isi_gather_info?

Jump to solution

To add to Ed's comment and answer William, I asked Support about removing old info from /ifs/.ifsvar/modules/jobengine/cp/ and was told that it's ok to do.

Our isi_gathers were running 3+ hours on a 12 node NL400 cluster. The /ifs/.ifsvar/modules/jobengine/cp/ directory was ~128000 folders and ~209000 files. Imagine manually tarring and gzipping that and you can see why it was taking so long.

After whittling this directory down from 1445 days worth of logs to 30, the isi_gather_info now only takes ~30 minutes!!

Apologies for reviving such an old post, but there isn't much out there regarding this, so I'm hoping this will help anyone that comes across this post.

Highlighted
2 Iron

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Please refer to Article Number 000459277

isi_gather_info gathers unexpectedly large logset

https://emcservice.force.com/CustomersPartners/kA2j0000000R3uoCAC

0 Kudos
Highlighted
2 Bronze

Re: How can I reduce the size of isi_gather_info?

Jump to solution

Also replying on a very old thread - our space usage looked to be somewhere entirely different, namely:

-rw-rw---- 1 root wheel 452M Mar 20 15:50 celog_ifsvar_db.tar
-rw-rw---- 1 root wheel 1.6G Mar 20 16:35 ifsvar_modules.tar

The top one is the event log DB, which is OK, as it's only a few files. The second one is worse, as it consists of:

# du -sh /ifs/.ifsvar/modules/tsm/*
174M /ifs/.ifsvar/modules/tsm/backup
4.5G /ifs/.ifsvar/modules/tsm/bwt
48M /ifs/.ifsvar/modules/tsm/config
64K /ifs/.ifsvar/modules/tsm/conflict_logs
64K /ifs/.ifsvar/modules/tsm/failpoint_files
8.0M /ifs/.ifsvar/modules/tsm/running
27M /ifs/.ifsvar/modules/tsm/samba-bak
9.1G /ifs/.ifsvar/modules/tsm/sched
73M /ifs/.ifsvar/modules/tsm/sched_edit
45M /ifs/.ifsvar/modules/tsm/target_work_locks
3.1M /ifs/.ifsvar/modules/tsm/work_locks

/ifs/.ifsvar/modules/tsm/bwt is 4.5GiB of low-level logs going back 6 years!

 

0 Kudos
Highlighted
2 Bronze

Re: How can I reduce the size of isi_gather_info?

Jump to solution

/ifs/.ifsvar/modules/tsm/sched/reports

looks to be just as bad...

 

0 Kudos