NetWorker: authc commands on a clustered Red Hat server report "unable to find valid certification path to the requested target."

Summary: NetWorker has been installed on a RedHat/CentOS Linux cluster using High Availability. When running authc config commands (authc_config, authc_mgmt) the command returns "unable to find a valid certification path to the requested target." ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • The NetWorker server deployment is configured on Red Hat/CentOS 7.x or 8.x Linux servers using High Availability clustering. 
  • authc_config and authc_mgmt commands return a certificate path error:
root@NWrhelNodeF:~# pcs resource
  * Resource Group: NW_group:
    * fs        (ocf::heartbeat:Filesystem):     Started NWrhelNodeF.emclab.local
    * ip        (ocf::heartbeat:IPaddr):         Started NWrhelNodeF.emclab.local
    * nws       (ocf::EMC_NetWorker:Server):     Started NWrhelNodeF.emclab.local

root@NWrhelNodeF:~# authc_mgmt -u Administrator -p 'authc_password' -e find-all-users
ERROR [main] (DefaultLogger.java:222) - Error while performing Operation:
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:9090/auth-server/api/v1/sec/authenticate": sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  • After a node failover occurs, there is a mismatch between the certificates in the shared /nsr location and the local /opt/nsr location on the new active node:
root@NWrhelNodeF:~# cd /opt/nre/java/latest/bin
root@NWrhelNodeF:/opt/nre/java/latest/bin#  ./keytool -list -keystore /nsr_share/nsr/authc/conf/authc.keystore -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin#  ./keytool -list -keystore /nsr/authc/conf/authc.keystore  -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin#  ./keytool -list -keystore /opt/nsr/authc-server/conf/authc.truststore  -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Apr 13, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 1C:32:BF:11:70:93:4E:DF:F5:77:42:DA:98:E5:5A:AF:FC:BB:9A:C6:8D:40:54:6E:77:9D:E2:2F:7D:50:C0:CD
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin#  ./keytool -list -keystore ../lib/security/cacerts -storepass changeit | grep -A1 emcauthctomcat
emcauthctomcat, Jan 31, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 1C:32:BF:11:70:93:4E:DF:F5:77:42:DA:98:E5:5A:AF:FC:BB:9A:C6:8D:40:54:6E:77:9D:E2:2F:7D:50:C0:CD
root@NWrhelNodeF:/opt/nre/java/latest/bin#
NOTE: When a node is active, /nsr is symbolically linked to the shared path (example: /nsr_share/nsr). That is why in the above output the certificate matches when comparing the /nsr_share and /nsr outputs. This path is shared between nodes; however, the /opt/nsr and /opt/nre (Java) paths are local to each physical node. The certificate signatures between the "shared" and "local" certificates do not match.
  • When the other node is active, the certificates between "local" and "shared" paths match
root@NWrhelNodeE:~# pcs resource
  * Resource Group: NW_group:
    * fs        (ocf::heartbeat:Filesystem):     Started NWrhelNodeE.emclab.local
    * ip        (ocf::heartbeat:IPaddr):         Started NWrhelNodeE.emclab.local
    * nws       (ocf::EMC_NetWorker:Server):     Started NWrhelNodeE.emclab.local
root@NWrhelNodeE:~#
root@NWrhelNodeE:~# cd /opt/nre/java/latest/bin
root@NWrhelNodeE:/opt/nre/java/latest/bin#
root@NWrhelNodeE:/opt/nre/java/latest/bin#  ./keytool -list -keystore /nsr_share/nsr/authc/conf/authc.keystore -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeE:/opt/nre/java/latest/bin#
root@NWrhelNodeE:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr/authc/conf/authc.keystore  -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeE:/opt/nre/java/latest/bin#
root@NWrhelNodeE:/opt/nre/java/latest/bin# ./keytool -list -keystore /opt/nsr/authc-server/conf/authc.truststore  -storepass 'authc_password' | grep -A1 emcauthctomcat
emcauthctomcat, Apr 13, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeE:/opt/nre/java/latest/bin#
root@NWrhelNodeE:/opt/nre/java/latest/bin# ./keytool -list -keystore ../lib/security/cacerts -storepass changeit | grep -A1 emcauthctomcat
emcauthctomcat, Apr 13, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
NOTE: Specify the authc keystore password for the -storepass value. The authc keystore password is configured during initial setup when running the /opt/nsr/authc-server/scripts/authc_configure.sh script on each node.
  • On the node where all certificates match, the certificate path error is not observed:
root@NWrhelNodeE:~# authc_mgmt -u Administrator -p 'authc_password' -e find-all-users
The query returns 1 records.
User Id User Name
1000    administrator

Cause

Before NetWorker servers are added to the cluster using /usr/sbin/networker.cluster, they are configured as stand-alone servers. /opt/nsr/authc-server/scripts/authc_configure.sh is run post install, generating unique certificates per node. The certificates that are used in the shared location match from whichever node was the active node, where the clustered nws resource was initially started on. 
 

The /nsr directory is symbolically linked to the /nsr_share/nsr directory which is moved between nodes depending on whichever node is the current active node. The /opt/nsr/authc-server/conf/authc.truststore is local to each node and is not shared when a failover occurs. After a node failover, the /nsr/authc/conf/authc.keystore emcauthc certificate signatures matched with the /opt/nsr certificates on the initial node but not the current active node 

Resolution

Solution:

This issue has been fixed in the following NetWorker versions:
  • 19.8.0.4
  • 19.9.0.2
Upgrade to one of the listed NetWorker versions (or later). NetWorker packages can be downloaded from: https://www.dell.com/support/home/product-support/product/networker/drivers
 
NOTE: After NetWorker has been upgraded and the clustered nws resource has been brought up, you must rerun /opt/nsr/authc-server/scripts/authc_configure.sh on the node where the certificate mismatch was observed. This corrects the mismatch.
  
root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr_share/nsr/authc/conf/authc.keystore -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Aug 31, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 2A:10:32:F4:09:13:8E:26:2C:11:63:DE:42:EF:AB:75:EF:29:6D:11:82:75:32:B6:27:71:96:FF:9D:A4:53:48
root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr/authc/conf/authc.keystore  -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Aug 31, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 2A:10:32:F4:09:13:8E:26:2C:11:63:DE:42:EF:AB:75:EF:29:6D:11:82:75:32:B6:27:71:96:FF:9D:A4:53:48
root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /opt/nsr/authc-server/conf/authc.truststore  -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Oct 20, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 43:80:AC:4A:78:BC:CA:5A:9F:DB:DF:04:30:B3:D1:41:F4:78:31:F8:0E:93:06:5F:F7:D6:A0:5F:E3:86:6B:AA

root@lnx-node1:/opt/nre/java/latest/bin# /opt/nsr/authc-server/scripts/authc_configure.sh
Specify the directory where the Java Standard Edition Runtime Environment (JRE) software is installed [/opt/nre/java/latest]:
The installation process will install an Apache Tomcat instance.
For optimum security, EMC NetWorker Authentication Service will
use a non-root user (nsrtomcat) to start the Apache Tomcat instance.
If your system has special user security requirements, ensure that proper
operational permissions are granted to this non-root user (nsrtomcat).
Please refer to NetWorker Installation Guide.
WARNING: Port 9090 is already in use.
Do you wish to specify a different port number [y]? n
The Apache Tomcat will use "lnx-node1.amer.lan" as the host name.
The Apache Tomcat will use "9090" as the port number.
The NetWorker Authentication Service requires a keystore file to configure encryption and to provide SSL support.
EMC recommends that you specify a keystore password that has a minimum of six characters.
Do you want to use the existing keystore /nsr/authc/conf/authc.keystore [y]?
Specify password for the existing keystore:
The install will use the existing certificate "emcauthctomcat" for Apache Tomcat.
The install will use the existing certificate "emcauthcsaml" for Authentication Service.
The NetWorker Authentication Service defines automatically an administrator user account
named administrator in the NetWorker Authentication Service local database.
This account is specific to the administration of the NetWorker Authentication Service, and
is not related to other administrator accounts on this system.
********************************************************************************************
Password criteria: Minimum required characters - 9 and Maximum allowed characters - 126
Minimum [alphabetic - 2, Uppercase - 1, Lowercase - 1, Numeric - 1, Special character - 1]
********************************************************************************************
Specify an initial password for administrator:
Confirm the password:
Creating the installation log in /opt/nsr/authc-server/logs/install.log.
Performing initialization. Please wait...
The installation completed successfully.

root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr/authc/conf/authc.keystore  -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Aug 31, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 2A:10:32:F4:09:13:8E:26:2C:11:63:DE:42:EF:AB:75:EF:29:6D:11:82:75:32:B6:27:71:96:FF:9D:A4:53:48
root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /opt/nsr/authc-server/conf/authc.truststore  -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Oct 20, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 2A:10:32:F4:09:13:8E:26:2C:11:63:DE:42:EF:AB:75:EF:29:6D:11:82:75:32:B6:27:71:96:FF:9D:A4:53:48
root@lnx-node1:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr_share/nsr/authc/conf/authc.keystore -storepass 'AUTHC_PASSWORD' | grep -A1 emcauthctomcat
emcauthctomcat, Aug 31, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 2A:10:32:F4:09:13:8E:26:2C:11:63:DE:42:EF:AB:75:EF:29:6D:11:82:75:32:B6:27:71:96:FF:9D:A4:53:48

root@lnx-node1:~# authc_mgmt -u Administrator -e find-all-users
Enter password:
The query returns 1 records.
User Id User Name
1000    administrator
 

Workaround:

1. Make the node where the certificates match the active node in pcs. Examples of how to determine this are shown in the Symptoms field. 
2. Log in to the passive node (where certificates do not match). 
3. Use the nsrssltrust command to create a certificate file against the virtual cluster resource: 

nsrssltrust -u https://cluster-hostname:9090 -c certificate_file.cer

Example:

root@NWrhelNodeF:~# nsrssltrust -u https://NWrhelClusD.emclab.local:9090 -c emcauthctomcat.cer
Fetching server's CA certificate chain / server certificate (if CA is not available)...

Information of the cert chain received from SSL server:

        idx: 0
        subject: /C=US/ST=TX/L=Round Rock/O=DELL/OU=NetWorker/CN=NWrhelNodeE.emclab.local
        issuer: /C=US/ST=TX/L=Round Rock/O=DELL/OU=NetWorker/CN=NWrhelNodeE.emclab.local
        Validity Not Before: Dec 19 17:03:27 2022 GMT
        Validity Not After: Dec 13 17:03:27 2047 GMT
        finger print sha1: 5d31f1a7bb4f3982f213235372503e3835c048e1
        signing algorithm: 1020

Do you trust this certificate(s) entity based on above information? [yes]/[no]:
yes
Https certificate is saved into certfile [emcauthctomcat.cer].

4. Confirm that the signature of the generated certificate matches the signature of the shared certificate on the active node:

cd /opt/nre/java/latest/bin
/opt/nre/java/latest/bin/keytool -printcert -file certificate_file.cer | grep SHA256

Example:

root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -printcert -file /root/emcauthctomcat.cer | grep SHA256
         SHA256: 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4

root@NWrhelNodeE:/opt/nre/java/latest/bin# ./keytool -printcert -file /nsr_share/nsr/authc/conf/emcauthctomcat.cer | grep SHA256
         SHA256: 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4

5. On the node with the mismatched certificates, delete the local emcauthctomcat certificates from the authc.truststore and cacerts file.

cd /opt/nre/java/latest/bin
./keytool -delete -alias emcauthctomcat -keystore /opt/nre/java/latest/lib/security/cacerts -storepass changeit
./keytool -delete -alias emcauthctomcat -keystore /opt/nsr/authc-server/conf/authc.truststore -storepass 'authc-password'
NOTE: If successful, no output is returned.

6. Import the certificate generated with nsrssltrust:

cd /opt/nre/java/latest/bin
./keytool -import -alias emcauthctomcat -keystore /opt/nsr/authc-server/conf/authc.truststore -storepass 'authc-password' -file certificate_file.cer
./keytool -import -alias emcauthctomcat -keystore /opt/nre/java/latest/lib/security/cacerts -storepass changeit -file certificate_file.cer
Example: 
root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -import -alias emcauthctomcat -keystore /opt/nsr/authc-server/conf/authc.truststore -storepass 'authc-password' -file /root/emcauthctomcat.cer
Owner: CN=NWrhelNodeE.emclab.local, OU=NetWorker, O=DELL, L=Round Rock, ST=TX, C=US
Issuer: CN=NWrhelNodeE.emclab.local, OU=NetWorker, O=DELL, L=Round Rock, ST=TX, C=US
Serial number: 6b0ed47e
Valid from: Mon Dec 19 12:03:27 EST 2022 until: Fri Dec 13 12:03:27 EST 2047
Certificate fingerprints:
         SHA1: 5D:31:F1:A7:BB:4F:39:82:F2:13:23:53:72:50:3E:38:35:C0:48:E1
         SHA256: 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
Signature algorithm name: SHA512withRSA
Subject Public Key Algorithm: 3072-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: localhost
  IPAddress: 127.0.0.1
  DNSName: NWrhelNodeE.emclab.local
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -import -alias emcauthctomcat -keystore /opt/nre/java/latest/lib/security/cacerts -storepass changeit -file /root/emcauthctomcat.cer
Certificate already exists in keystore under alias <emcnwuiserv>
Do you still want to add it? [no]:  yes
Certificate was added to keystore

Test:

To validate that the certificates are now valid, failover the cluster to the node the above changes were applied to:

1. The authc_mgmt or authc_config commands should now work on the node they were previously failing:

root@NWrhelNodeF:~# authc_mgmt -u Administrator -p 'NetWorker_Admin_password' -e find-all-users
The query returns 1 records.
User Id User Name
1000    administrator

2. For additional verification, review the certificates from both the local and shared locations:

root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr_share/nsr/authc/conf/authc.keystore -storepass 'authc-password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -list -keystore /nsr/authc/conf/authc.keystore  -storepass 'authc-password' | grep -A1 emcauthctomcat
emcauthctomcat, Dec 19, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -list -keystore /opt/nsr/authc-server/conf/authc.truststore  -storepass 'authc-password' | grep -A1 emcauthctomcat
emcauthctomcat, Apr 13, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4
root@NWrhelNodeF:/opt/nre/java/latest/bin#
root@NWrhelNodeF:/opt/nre/java/latest/bin# ./keytool -list -keystore ../lib/security/cacerts -storepass changeit | grep -A1 emcauthctomcat
emcauthctomcat, Apr 13, 2023, trustedCertEntry,
Certificate fingerprint (SHA-256): 4C:A0:07:70:D2:04:4D:A2:F0:87:5A:75:4A:3A:9F:C3:B6:3A:C0:3B:05:F8:9C:F6:81:01:D5:8C:C7:CE:E6:B4

Additional Information

If the AUTHC/NMC server is a central stand-alone NetWorker system, this issue does not affect NMC, NWUI, or REST API requests.
  • NetWorker Management Console (NMC)
  • NetWorker Web User Interface (NWUI)
  • NetWorker Representational State Transfer Application Programming Interface (REST API)
RESTAPI: How to use a remote AUTHC server when processing RESTAPI requests?
NetWorker: Server is configured on RHEL 8.x failover cluster fails to start

Affected Products

NetWorker

Products

NetWorker Family, NetWorker Series
Article Properties
Article Number: 000205383
Article Type: Solution
Last Modified: 24 Mar 2025
Version:  9
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.