VNX: FS /dev/hda3 mounted on / filling up (at 100%, max = 90%). (User Correctable)

The NAS root partition FS /dev/hda3 mounted on / is close to 100% full to capacity:

[nasadmin@system~]$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 2030768 1770484 155460 92% /

The /nas/log/sys_log files regularly report the following alerts:

Aug 21 08:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[32529]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 09:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[1798]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 10:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[482]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 11:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[5219]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 12:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[5950]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 13:03:02 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[6997]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 14:03:02 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[11988]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 15:03:01 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[21169]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%). Aug 21 16:03:02 2017:CS_PLATFORM:ADMIN:CRITICAL:24:::::dskMon[23129]: FS /dev/hda3 mounted on / filling up (at 92%, max = 90%).

To identify what is using up all of the space, a capacity analysis is required starting at the top root directory, working your way down through the underlying sub-directories looking for any unusually large files or folders. The culprit in this instance was the /var/log/secure files which were very large in size. 

[root@sysadmin log]# ls -l | grep secure
-rw------- 1 root root 532710387 Aug 21 16:18 secure
-rw------- 1 root root 516363636 Jun 29 16:30 secure.1
-rw------- 1 root root 509383821 Jun 29 16:30 secure.2
-rw------- 1 root root 501638221 Jun 29 16:30 secure.3
-rw------- 1 root root 503321757 Jun 29 16:30 secure.4

An analysis of the /var/log/secure files shows that there is a host with IP X.X.X.X logging in to the Control Station every few seconds. Each time this host logs in to the Control Station, this is logged in /var/log/secure file and over time, the size of the file is increases. In this instance, an SRM server on the clients network was used to poll the storage for Monitoring and Reporting purposes. Over an 8 hour period, there was 8,551 logins to the Primary Control Station from host IP X.X.X.X:

Aug 21 16:38:16 system sshd[10390]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:38:16 system sshd[10388]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:43:15 system sshd[21621]: Accepted password for srmfile from X.X.X.X port 42314 ssh2 Aug 21 16:43:15 system sshd[21621]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) Aug 21 16:43:15 system sshd[21625]: subsystem request for sftp Aug 21 16:43:15 system sshd[21622]: Accepted password for srmfile from X.X.X.X port 58440 ssh2 Aug 21 16:43:15 system sshd[21622]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) Aug 21 16:43:15 system sshd[21627]: subsystem request for sftp Aug 21 16:43:15 system sshd[21621]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:43:15 system sshd[21622]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:43:16 system sshd[21641]: Accepted password for srmfile from X.X.X.X port 51964 ssh2 Aug 21 16:43:16 system sshd[21641]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) Aug 21 16:43:16 system sshd[21645]: subsystem request for sftp Aug 21 16:43:16 system sshd[21643]: Accepted password for srmfile from X.X.X.X port 60183 ssh2 Aug 21 16:43:16 system sshd[21643]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) Aug 21 16:43:16 system sshd[21648]: subsystem request for sftp Aug 21 16:43:16 system sshd[21641]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:43:16 system sshd[21643]: pam_unix(sshd:session): session closed for user srmfile Aug 21 16:43:16 system sshd[21650]: Accepted password for srmfile from X.X.X.X port 53097 ssh2 Aug 21 16:43:16 system sshd[21650]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) Aug 21 16:43:16 system sshd[21654]: subsystem request for sftp Aug 21 16:43:16 system sshd[21652]: Accepted password for srmfile from X.X.X.X port 47603 ssh2 Aug 21 16:43:16 system sshd[21652]: pam_unix(sshd:session): session opened for user srmfile by (uid=0) [root@system log]# more secure | grep " Accepted password for srmfile from X.X.X.X" | wc -l 8551 < -------- 8,551 logins

The size of the /var/log/secure files may not be increasing aggressively, a 1% - 2% increase may be experienced each day, however this will accumulate over time and cause the used capacity of /dev/hda3 to exceed the 90% threshold. To resolve this issue, a number options are available:

Option 1: Disable SRM to stop the aggressive logging. 

Option 2: Reduce the amount of SRM polling to the Control Station. 

Option 3:  Do nothing and proactively clean the /var/log/secure file approximately once every month depending on how quickly the files are increasing:

[root@system log]# echo > secure
[root@system log]# echo > secure.1
[root@system log]# echo > secure.2
[root@system log]# echo > secure.3
[root@system log]# echo > secure.4

[root@system log]# ls -l | grep secure
-rw------- 1 root root 1 Aug 21 16:18 secure
-rw------- 1 root root 0 Jun 29 16:30 secure.1
-rw------- 1 root root 0 Jun 29 16:30 secure.2
-rw------- 1 root root 0 Jun 29 16:30 secure.3
-rw------- 1 root root 0 Jun 29 16:30 secure.4