NetWorker: NetWorker REST API emcauthctomcat certificate removed from cacerts keystore

-- Can happen post upgrade of NetWorker to 9.2.1.X or after upgrading the RHEL OS.

-- emcauthtomcat certificate consistently disappears from the cacerts / java keystore.
-- As a result, the NetWorker REST API calls intermittently fail due to the fact the emcauthtomcat certicate is not in the cacerts keystore.
-- It can be imported manually (see Notes section below for details) and it works for some time.
-- But eventually it will go missing again (After approx 30 mins -> 1 hour).
-- As a workaround,  the certificate removal can be stopped by making the cacerts keystore "immutable" at the OS level.

Example Error Message:

# curl -X POST -H "Content-Type: application/json" -H "Authorization:Basic Ok5ldHckkkoiuol" -d '{"hostname":" host123"}' "https://host123t:9090/nwrestapi/v2/global/clients" -k -1 {"message":"javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://localhost:9090/auth-server/api/v1/sec/authenticate: java.security.cert.CertificateException: No X509TrustManager implementation available","status":{"code":500,"codeClass":"Server Error","reasonPhrase":"Internal Server Error"},"timestamp":"2018-08-28T09:47:49Z","userAgentRequest":{"contentLength":27,"headers":[{"name":"Accept","value":"*/*"},{"name":"Content-Type","value":"application/json"}],"method":"POST","url":"https://host123:9090/nwrestapi/v2/global/clients"},"version":"9.2.1.4.Build.233"}

-- SSL handshake failure between NetWorker REST API & NetWorker Authentication server due to the certificate being absent from the keystore.

-- Import the certificate using the steps outlined in the notes section below.
-- If issues persists, make the cacerts keystore immutable & contact EMC for the hotfix outlined under the reference ESC 32640 / BUG 301682.
-- Once patch has been applied, make keystore mutable again and re-import the certificate.

How it works... 
-- NetWorker stores the certificate in here: /opt/nsr/authc-server/conf which it gets from the java install here: %JAVA_HOME%\lib\security\cacerts 
-- NetWorker stores the authentication certificates in the file name auth.truststore.
-- For Linux installations, this file will be stored in the following path, /opt/nsr/authc-server/conf and whenever a client process requires certificate, it will look into this path. 

How to import the certificate...(Linux)
-- To find out the exact current Java version, start with the following command: rpm -qa | egrep 'jre|jdk'
-- Output shows package name installed. Ex: jre1.8.0_66-1.8.0_66-fcs  
-- (Note you may have 2/3 packages installed... run the below command against each of them until you get the location of the cacerts keystore)
-- Then run following command against package name. Ex: rpm -qVv jre1.8.0_66-1.8.0_66-fcs | grep cert
-- Output shows location of required file: /usr/java/jre1.8.0_66/lib/security/cacerts

-- Change to the Java JRE installation directory with the binaries. 
-- The directory will change depending on the installed Java JRE version. Ex: /usr/java/jre1.8.0_151/bin
-- Validate if the certificate does not exist in the Java JRE cacerts keystore: keytool -storepass changeit -list -keystore /usr/java/jre1.8.0_66/lib/security/cacerts |grep emc
-- Import the emcauthctomcat certificate if it's not there: keytool -importcert -storepass changeit -trustcacerts -alias emcauthctomcat -file /nsr/authc/conf/emcauthctomcat.cer -keystore /usr/java/jre1.8.0_66/lib/security/cacerts
-- Then validate if the certificate now exists in the Java JRE cacerts keystore: keytool -storepass changeit -list -keystore /usr/java/jre1.8.0_66/lib/security/cacerts | grep emc