Isilon node /var file system is almost full

Summary: Isilon node /var file system shows more than 75% usage, yet no files are found that occupy the space.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Unable to do OneFS upgrade due to lack of space on /var file system on one of the nodes.

The node shows the following usage of the file systems: 
# df -h
Filesystem               Size    Used   Avail Capacity  Mounted on
/dev/mirror/root0        956M    475M    404M    54%    /
devfs                    1.0K    1.0K      0B   100%    /dev
/dev/mirror/var0         956M    729M    151M    83%    /var
/dev/mirror/var-crash    1.9G     13M    1.7G     1%    /var/crash
/dev/mirror/keystore      31M    4.0K     28M     0%    /keystore
/dev/md0                  60M    1.1M     54M     2%    /tmp/ufp
/dev/md1.uzip            106M    101M   -2.9M   103%    /base
OneFS                    162T    122T     34T    78%    /ifs

# du -skx /var/log/* |sort -nk 1
...
...
1824    /var/log/isi_papi_d.log
2032    /var/log/ConnectEMCHttp_00.log
2032    /var/log/ConnectEMCHttp_01.log
2032    /var/log/ConnectEMCHttp_02.log
2032    /var/log/ConnectEMC_01.log
2032    /var/log/ConnectEMC_02.log
2032    /var/log/ConnectEMC_03.log
2032    /var/log/ConnectEMC_04.log
2032    /var/log/ConnectEMC_05.log
2032    /var/log/ConnectEMC_06.log
2032    /var/log/ConnectEMC_07.log
2192    /var/log/lsassd.log
2288    /var/log/hw_ata.log
2432    /var/log/isi_radish.log
2764    /var/log/apache2
4640    /var/log/isi_km_d.log
4736    /var/log/isi_celog_analysis.log
6992    /var/log/isi_celog_events.log
15760   /var/log/nfs.log
24928   /var/log/messages
43824   /var/log/lwiod.log

Used space is more than 700 Mbytes yet cannot find files that have more than 200 Mbytes: 
# du -sh /var/crash
 13M    /var/crash

# du -sh /var
201M    /var

Tried to unmount /var/crash and looked whether there are files under /var/crash but nothing is found there.

Cause

Orphaned files that were deleted but space was not reclaimed.

Resolution

Reboot the node to reclaim the space. After reboot, the following file system usage is present: 
# df -h
Filesystem               Size    Used   Avail Capacity  Mounted on
/dev/mirror/root0        956M    475M    404M    54%    /
devfs                    1.0K    1.0K      0B   100%    /dev
/dev/mirror/var0         956M    195M    684M    22%    /var
/dev/mirror/var-crash    1.9G     13M    1.7G     1%    /var/crash
/dev/mirror/keystore      31M    4.0K     28M     0%    /keystore
/dev/md0                  60M    836K     54M     1%    /tmp/ufp
/dev/md1.uzip            106M    101M   -2.9M   103%    /base
OneFS                    162T    124T     33T    79%    /ifs

Affected Products

Isilon

Products

Isilon
Article Properties
Article Number: 000173032
Article Type: Solution
Last Modified: 15 Apr 2022
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.