VBA: Var file system has no available space
Summary: The NetWorker VMware Backup Appliance's (VBA) /var file system is full.
Symptoms
The VBA /var file system is full.
admin@VBA-Name:~/>: df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.9G 5.7G 1.8G 77% /
udev 3.9G 136K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 130M 38M 86M 31% /boot
/dev/sda7 1.5G 1.4G 0 100% /var
/dev/sda9 70G 13G 54G 20% /space
/dev/sdb1 256G 20G 237G 8% /data01
/dev/sdd1 256G 13G 244G 5% /data02
/dev/sdc1 256G 12G 245G 5% /data03
Cause
Tomcat listens to port 9443 and also, whenever tomcat service does not run and simultaneously when anyone tries to access tomcat services, the errors regarding tomcat not running gets flooded in /var/log/apache2/mod_jk.log.
Note: The default web services configuration does not limit the number of logs retained or the size of each log. In environments with large activity the default tomcat web services configuration may lead to the var file system becoming full.
Resolution
Perform the following on the vProxy appliance.
1. Open an SSH session to the VBA, log in as admin, then switch to root:
2. Change the directories to the apache2 directory:
cd /var/log/apache2/
3. Confirm disk space usage with the following du command:
root@ VBA-Name:/var/log/apache2/#: du -sh
1.3G
4. Run the following ls command to review the file size and disk usage:
root@ VBA-Name:/var/log/apache2/#: ls -lhtr
total 1.3G
-rw-r----- 1 root root 53K Aug 28 2016 access_log-20160828.bz2
-rw-r----- 1 root root 27K Sep 14 2016 ssl_request_log-20160914.bz2
-rw-r----- 1 root root 30K Oct 14 2016 ssl_request_log-20161014.bz2
-rw-r----- 1 root root 56K Nov 3 2016 access_log-20161103.bz2
-rw-r----- 1 root root 25K Nov 13 2016 ssl_request_log-20161113.bz2
-rw-r----- 1 root root 27K Dec 13 2016 ssl_request_log-20161213.bz2
-rw-r----- 1 root root 51K Jan 9 2017 access_log-20170109.bz2
-rw-r----- 1 root root 26K Jan 12 2017 ssl_request_log-20170112.bz2
-rw-r----- 1 root root 26K Feb 11 2017 ssl_request_log-20170211.bz2
-rw-r----- 1 root root 30K Mar 13 2017 ssl_request_log-20170313.bz2
-rw-r----- 1 root root 56K Mar 17 2017 access_log-20170317.bz2
-rw-r----- 1 root root 28K Apr 11 2017 ssl_request_log-20170411.bz2
-rw-r----- 1 root root 28K May 11 12:30 ssl_request_log-20170511.bz2
-rw-r----- 1 root root 54K May 23 12:30 access_log-20170523.bz2
-rw-r----- 1 root root 39K Jun 7 12:30 ssl_request_log-20170607.bz2
-rw-r----- 1 root root 31K Jul 6 12:30 ssl_request_log-20170706.bz2
-rw-r----- 1 root root 70K Jul 22 12:30 access_log-20170722.bz2
-rw-r----- 1 root root 29K Aug 4 12:30 ssl_request_log-20170804.bz2
-rw-r----- 1 root root 30K Sep 1 12:30 ssl_request_log-20170901.bz2
-rw------- 1 root root 10 Oct 16 02:43 rcapache2.out
-rw-r----- 1 root root 2.6M Nov 7 10:46 ssl_request_log
-rw------- 1 root root 1.2G Nov 7 10:46 mod_jk.log
-rw------- 1 root root 292K Nov 7 10:46 error_log
-rw-r----- 1 root root 3.8M Nov 7 10:46 access_log
5. Stop services:
root@ VBA-Name:/var/log/apache2/#: website stop
===Shutting down website
Shutting down httpd2 (waiting for all children to terminate) done
root@ VBA-Name:/var/log/apache2/#: rm mod_jk.log
7. Start the website service:
root@ VBA-Name:/var/log/apache2/#: website start
===Starting website
Starting httpd2 (prefork) done
8. Confirm that that disk space usage has reduced:
root@ VBA-Name:/var/log/apache2/#: df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.9G 5.7G 1.8G 77% /
udev 3.9G 136K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 130M 38M 86M 31% /boot
/dev/sda7 1.5G 172M 1.2G 13% /var
/dev/sda9 70G 13G 54G 20% /space
/dev/sdb1 256G 21G 235G 9% /data01
/dev/sdd1 256G 13G 244G 5% /data02
/dev/sdc1 256G 12G 245G 5% /data03
9. Verify if the new mod_jk.log has been generated.
root@ VBA-Name:/var/log/apache2/#: ls -lhtr | grep -i mod_jk.log
-rw------- 1 root root 600 Nov 8 14:32 mod_jk.log