Avamar: No space left on device error due to the number of inodes

Running the logrotate error generates an error indicating that there is no space:

logrotate -f /etc/logrotate.conf

error: error creating output file /var/log/firewall: No space left on device 
error: error creating output file /var/log/mail.err: No space left on device 
error: error creating output file /var/log/wtmp-nnnnnnnn: No space left on device 

The df command shows available space:

df -h

Filesystem     Size  Used  Avail  Use%  Mounted on 
/dev/sda5      7.9G  4.2G   3.4G   56%  / 
devtmpfs       5.8G  248K   5.8G    1%  /dev 
tmpfs          5.8G     0   5.8G    0%  /dev/shm 
/dev/sda1      114M   53M    56M   49%  /boot 
/dev/sda3      256G  19G    237G    8%  /data01 
/dev/sda7      1.5G  524M   899M   37%  /var 

Attempting to create a file on the /var file system also results in a space error:

touch /var/log/test

touch: cannot touch '/var/log/test': No space left on device

The file system has run out of inodes.

Rerunning the df command with the "-i" switch can confirm this:

df -i 

Filesystem      Inodes   IUsed     IFree  IUse%  Mounted on 
/dev/sda5       524288  106337    417951    21%  / 
devtmpfs             0      0          0     -   /dev 
tmpfs          1513794      1    1513793     1%  /dev/shm 
/dev/sda1        30120     49      30071     1%  /boot 
/dev/sda3    267691072   5438  267685634     1%  /data01 
/dev/sda7        97536  97536          0   100%  /var

This is generally caused by either a fragmented file system, lots of very small files, temp files, or a large mail queue.

The du command can be run to see what is taking up the space:

du -x -h /var --max-depth=1 

Example output showing that the /var/spool directory takes up most of the space within /var:

8.0K    /var/state 
4.0K    /var/X11R6 
4.0K    /var/crash 
45M     /var/lib 
4.0K    /var/named 
4.0K    /var/opt 
12K     /var/yp 
380M    /var/spool 
132K    /var/run 
4.0M    /var/cache 
25M     /var/adm 
16K     /var/lost+found 
8.0K    /var/tmp 
28K     /var/lock 
37M     /var/log 
56K     /var/games 
489M    /var

The du command can then be run again further down the directory tree:

du -x -h /var/spool --max-depth=1

Further investigation is required depending on what is taking up the space.

If assistance is required, open a Service Request (SR) with the Dell Technologies Avamar Support team.

In this example: the issue was many very small files.

• • • • The largest directory was /var/spool/postfix/maildrop, and contained thousands of small files:

ls -l /var/spool/postfix/maildrop| wc -l 

96559

• • • • Each of the files contained the following:

admin : /etc/sudoers is mode 0777, should be 0440

• • • • The permissions and ownership for the sudoers file were updated:

chmod 0440 /etc sudoers
chown root:root /etc/sudoers

• • • • The files in the /var/spool/postfix/maildrop directory were removed, and the issue was resolved.