Dell VNX: Cannot access Unisphere UI Web Interface after upgrade

Flare OE and NAS upgrades.

Unable to access the Unisphere Web Interface after an OE Flare and NAS code upgrade. Unisphere became inaccessible, the httpd.conf file reverted back to default and upon editing an extra space was entered causing problems with the Apache script.

Customer had to reedit the /nas/http/conf/httpd.conf after an upgrade as the file reverts back to default settings.
In this case, the customer had to readd security Ciphers on the SSLCipherSuite line. However, any mistake in editing such as leaving an extra space in the file can cause the Apache script to fail.

To confirm a problem with the httpd.conf file, check error instances regarding Apache and Tomcat 'unexpectedly exited' using the following commands:

/nas/tools/dbchk -wvxpV
nas_logviewer /nas/log/sys_log | grep -i apache | tail
nas_logviewer /nas/log/sys_log | grep -i tomcat | tail
cat /var/log/messages | grep -i tomcat | tail
cat /var/log/messages | grep -i apache | tail
cat /nas/tomcat/logs/catalina.out  | grep -i error | tail

Check for syntax errors in the apache_restart.out file using the following command:

cat /nas/http/logs/apache_restart.out | grep -i syntax | tail

To confirm issues with the httpd.conf:
Do a vi or less on /nas/http/conf/httpd.conf file and search for the line number that you see w/ the syntax error from apache_restart.out. And then check the same file and line number on a lab array to determine where the extra space or incorrect/missing character is and edit to resolve.

After confirming an edit issue with the httpd.conf file:
For the SSLCipherSuite example, you would vi edit and remove the extra space on that line to change it from one line broken into two lines back to a single line:

< # SSL Cipher Suite:
< # List the ciphers that the client is permitted to negotiate.
< # See the mod_ssl documentation for a complete list.
< #SSLCipherSuite ALL:!ADH:!DH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:-MEDIUM:-LOW
< SSLCipherSuite
< ALL:!ADH:!DH:!EXPORT:!SSLv2:+HIGH:-MEDIUM:-LOW

After removing the extra space, it would then look like this:

< # SSL Cipher Suite:
< # List the ciphers that the client is permitted to negotiate.
< # See the mod_ssl documentation for a complete list.
< #SSLCipherSuite ALL:!ADH:!DH:!EXPORT:!SSLv2:+HIGH:-MEDIUM:-LOW
< SSLCipherSuite ALL:!ADH:!DH:!EXPORT:!SSLv2:+HIGH:-MEDIUM:-LOW

After fixing the editing mistake in the httpd.conf file, confirm that the errors have stopped by reviewing the logs again with:

tail -f on /var/log/messages

And

/nas/http/logs/apache_restart.out

The errors tend to be streaming and stop right after you fix the problem edit.
Remember: The above is an example to show a space added to a single line breaking it up and causing it to be two separate lines in the file. But you would have to assess on a case by case basis and fix accordingly be it extra spaces, removed spaces, or typos.

From /nas/log/syslog:

# nas_logviewer /nas/log/sys_log | grep -i apache | tail -5

Sep 11 11:48:25 2019:CS_PLATFORM:MasterControl:EMERGENCY:6:::::Daemon Apache daemon unexpectedly exited (status = 0); ifexit=1, exitstatus=0, ifsignal=0, termsig=0, ifstop=0, stopsig=0, ifdump=0.
Sep 11 11:48:25 2019:CS_PLATFORM:MasterControl:EMERGENCY:6:::::Daemon Apache daemon unexpectedly exited (status = 0); ifexit=1, exitstatus=0, ifsignal=0, termsig=0, ifstop=0, stopsig=0, ifdump=0.
Sep 11 11:48:25 2019:CS_PLATFORM:MasterControl:EMERGENCY:6:::::Daemon Apache daemon unexpectedly exited (status = 0); ifexit=1, exitstatus=0, ifsignal=0, termsig=0, ifstop=0, stopsig=0, ifdump=0.
Sep 11 11:48:26 2019:CS_PLATFORM:MasterControl:EMERGENCY:6:::::Daemon Apache daemon unexpectedly exited (status = 0); ifexit=1, exitstatus=0, ifsignal=0, termsig=0, ifstop=0, stopsig=0, ifdump=0.
Sep 11 11:48:26 2019:CS_PLATFORM:MasterControl:EMERGENCY:15:::::Apache daemon respawning too fast; disabled for 5 minutes.

From /nas/tomcat/logs/catalina.out: You can see an "SEVERE: Error decoding request" message.

Aug 28, 2019 8:23:31 PM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Aug 28, 2019 8:23:31 PM org.apache.jk.common.HandlerRequest invoke
SEVERE: Error decoding request
java.io.CharConversionException: Invalid char in port: 58
at org.apache.jk.common.HandlerRequest.parseHost(HandlerRequest.java:658)
at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:404)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:261)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Unknown Source)
Aug 28, 2019 8:23:31 PM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Wed Sep 4 10:24:48 CDT 2019 Starting tomcat web server.

If for any reason a restart of Tomcat, Apache and httpd is needed after confirming the httpd.conf file has no further issues, you can do so using this command:

/nas/http/nas_ezadm/etc/script restart

It should be noted that restarting Apache/Tomcat/httpd alone has no effect if this is a confirmed issue or mistake in vi editing of the file, until after the edit mistake is corrected. If the httpd.conf file has any bad syntax, you see "SEVERE: Catalina.stop: java.net.ConnectException: Connection refused (Connection refused)" error that would show up during restart of services.

Also, you can check a regular getagent command:

/nas/sbin/naviseccli -h SPA getagent

And then, also check security creds to be sure that there are no issues:

# /nas/sbin/naviseccli -h SPA -user sysadmin -password sysadmin -scope 0 getagent

Agent Rev: 7.33.9 (2.36)
Name: K10
Desc:
Node: A-APM00xxxxxxxxx
Physical Node: K10
Signature: 3854449
Peer Signature: 3698693
Revision: 05.33.009.5.238
SCSI Id: 0
Model: VNX7600
Model Type: Rackmount
Prom Rev: 33.51.00
SP Memory: 65536
Serial No: APM00xxxxxxxxx
SP Identifier: A
Cabinet: DPE9

If security naviseccli command fails do a KBA search using the following to find several related KB Articles:

VNX:   /nas/sbin/naviseccli -h SPA -user <user> -password <password> -scope 0 getagent