Isilon Process httpd of service isi_webui has failed to restart after multiple attempts - Event ID: 400030001

The OneFS WebUI is inaccessible if the webui_httpd.conf file under /var/run/apache2/ became stale on the problematic node:

isilon-1# ps axuww | grep httpd | grep -v grep
root   43633   0.0  0.5 170644 10188  -  Ss    2:49PM     0:00.06 /usr/local/apache2/bin/httpd -k start
daemon 43634   0.0  0.5 170332  9840  -  I     2:49PM     0:00.00 /usr/local/apache2/bin/httpd -k start
daemon 43638   0.0  0.6 174360 11652  -  I     2:49PM     0:00.00 /usr/local/apache2/bin/httpd -k start
<<< the output is missing "/usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start" thread. There should be 3 or more threads on each node

Attempting to manually start httpd of isi_webui service would result in the following message:

isilon-1# /usr/local/sbin/webui_httpd_ctl start
httpd: Could not reliably determine the server's fully qualified domain name, using 27.27.27.102 for ServerName
httpd (pid 72923) already running

isilon-1# cat /var/apache2/run/webui_httpd.pid
72923

If a node is restarted unexpectedly or if the webui process is killed in certain ways, the pid files are left. When either MCP or a node reboot restarts webui, webui may not start correctly, emitting the error:

> httpd (pid 72923) already running

(where 72923 was the pid of the previously-running webui daemon)

SSH to the problematic node with "root" account
Verify if webui_httpd.pid exists:

# ls -l /var/apache2/run/webui_httpd.pid

If it does, remove that .pid file with the below command:

# cd /var/apache2/run/; rm -f webui_httpd.pid

- Attempt to start httpd again with:

# /usr/local/sbin/webui_httpd_ctl start

After following the steps above, the output should be similar to the below where there should be three or more httpd of webui threads:

jv8004-1# ps axuww | grep "webui_httpd.conf" | grep -v grep
root    3135   0.0  0.6 261704 13280  -  Ss    4Apr18     0:18.62 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start
daemon  3140   0.0  0.6 261668 13268  -  I@    4Apr18     0:00.23 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start
daemon  3145   0.0  1.5 307568 30924  -  I     4Apr18     0:10.58 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start

Try to log in to the web administration on the problematic node after the above steps are taken.