Isilon: Apache HTTP logging can lead to premature boot drive failure due to excessive writes

Apache HTTP logging can lead to premature boot drive failure due to excessive writes.

By default, Apache is configured to log all HTTP access and HTTP errors events to the following logs:

/var/log/apache2/webui_httpd_error.log
/var/log/apache2/webui_httpd_access.log

Depending on the workflow, this logging can lead to significant wear on the boot drives causing premature failure.

Disable or decrease the amount of logging to the webui_httpd_error.log and webui_httpd_access.log by modifying the syslogd log level.

1. Back up the existing syslog configuration.

   cp /etc/mcp/templates/syslog.conf /ifs/data/Isilon_Support/syslog.conf <-Copy for editing
   cp /etc/mcp/templates/syslog.conf /ifs/data/Isilon_Support/syslog.conf.old <-Copy of original

2. Modify syslog.conf with vi, nano, or similar text editor.

 nano /ifs/data/Isilon_Support/syslog.conf

3. Locate the section !httpd it appears as follows:

   !httpd
   local2.*                                        /var/log/apache2/webui_httpd_error.log
   local3.*                                        /var/log/apache2/webui_httpd_access.log  

4. Comment out the following lines with a statement including the SR number (if any) and why this was modified for so it will be clear in the future why this change was made.

   #COMMENT OUT FOR SR <SR NUMBER> TO REDUCE APACHE LOGGING
   #!httpd
   #local2.*                                        /var/log/apache2/webui_httpd_error.log
   #local3.*                                        /var/log/apache2/webui_httpd_access.log  

 5. Add the following lines with a statement including the SR number (if any) and indicating the log level is set to none.
   #LOG LEVEL CHANGED TO NONE FOR SR <SR NUMBER>
   !httpd
   local2.none                                      /var/log/apache2/webui_httpd_error.log
   local3.none                                      /var/log/apache2/webui_httpd_access.log

   #LOG LEVEL CHANGED TO NONE FOR SR <SR NUMBER>
   !httpd
   local2.none                                      /var/log/apache2/webui_httpd_error.log
   local3.none                                      /var/log/apache2/webui_httpd_access.log 

4. Copy the edited file to all nodes.

 isi_for_array 'cp /ifs/data/Isilon_Support/syslog.conf /etc/mcp/templates/syslog.conf'

5. Verify that the new file has been copied successfully to all nodes.

   Isilon02-1# md5 /ifs/data/Isilon_Support/syslog.conf
   MD5 (/ifs/data/Isilon_Support/syslog.conf) = f0626c10f6a88dabeb13183255debba4
   Isilon02-1# isi_for_array -s 'md5 /etc/mcp/templates/syslog.conf'
   Isilon02-1: MD5 (/etc/mcp/templates/syslog.conf) = f0626c10f6a88dabeb13183255debba4
   Isilon02-2: MD5 (/etc/mcp/templates/syslog.conf) = f0626c10f6a88dabeb13183255debba4
   Isilon02-3: MD5 (/etc/mcp/templates/syslog.conf) = f0626c10f6a88dabeb13183255debba4 

6. Once copied restart the syslog for change to take effect.
  

isi_for_array 'killall -HUP syslogd'

7. Confirm logging to the access and error logs have stopped and are no longer spamming or rolling over frequently.

   tail -f /var/log/apache2/webui_httpd_access.log
   tail -f /var/log/apache2/webui_httpd_error.log