Symptoms
The OneFS WebUI will be inaccessible if the webui_httpd.conf file under /var/run/apache2/ became stale on the problematic node:
isilon-1# ps axuww | grep httpd | grep -v grep
root 43633 0.0 0.5 170644 10188 - Ss 2:49PM 0:00.06 /usr/local/apache2/bin/httpd -k start
daemon 43634 0.0 0.5 170332 9840 - I 2:49PM 0:00.00 /usr/local/apache2/bin/httpd -k start
daemon 43638 0.0 0.6 174360 11652 - I 2:49PM 0:00.00 /usr/local/apache2/bin/httpd -k start
<<< the output is missing "/usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start" thread. There should be 3 or more threads on each node
Attempting to manually start httpd of isi_webui service would result in the following message:
isilon-1# /usr/local/sbin/webui_httpd_ctl start
httpd: Could not reliably determine the server's fully qualified domain name, using 27.27.27.102 for ServerName
httpd (pid 72923) already running
isilon-1# cat /var/apache2/run/webui_httpd.pid
72923
Cause
Resolution
SSH to the problematic node with "root" account.
Verify if webui_httpd.pid exists:
# ls -l /var/apache2/run/webui_httpd.pid
If it does, remove that .pid file with the below command:
# cd /var/apache2/run/; rm -f webui_httpd.pid
- Attempt to start httpd again with
# /usr/local/sbin/webui_httpd_ctl start
After following the steps above, the output should be similar to the below where there should be 3 or more httpd of webui threads:
jv8004-1# ps axuww | grep "webui_httpd.conf" | grep -v grep
root 3135 0.0 0.6 261704 13280 - Ss 4Apr18 0:18.62 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start
daemon 3140 0.0 0.6 261668 13268 - I@ 4Apr18 0:00.23 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start
daemon 3145 0.0 1.5 307568 30924 - I 4Apr18 0:10.58 /usr/local/apache2/bin/httpd -f /usr/local/apache2/conf/webui_httpd.conf -k start
Try to log in to the web administration on problematic node after the above steps are taken.