NVP vProxy: All VMware backups failing "curl_easy_perform" returned error 60: "SSL certificate"
Summary: All NetWorker VMware Protection (NVP) Virtual Machine (VM) backups are failing. The job reports that the certificate used by the vProxy is incorrect.
Symptoms
All VM backups fail with the following error in the action logs:
MM/DD/YY HH:mm:SS Action backup vmware-vproxy 'ACTION_NAME' has initialized as 'vproxysave job' with job id 2080030
MM/DD/YY HH:mm:SS Starting nsrvim.
MM/DD/YY HH:mm:SS Calling the nsrvim program to collect the inventory data.
MM/DD/YY HH:mm:SS Setting default timeout 1800.
MM/DD/YY HH:mm:SS Using a timeout of 1800 seconds for the nsrvim request. Minimum timeout is 360 seconds. Maximum timeout is 3600 seconds.
MM/DD/YY HH:mm:SS Getting list of VM work items.
MM/DD/YY HH:mm:SS Transport mode selected from the user for backup: 'Auto'.
MM/DD/YY HH:mm:SS Saving the backup data in the pool 'POOL_NAME'.
MM/DD/YY HH:mm:SS Received the media management binding information on the host 'NW_SERVER_NAME'.
MM/DD/YY HH:mm:SS Connected to the nsrmmd process on the host 'NW_SERVER_NAME'.
MM/DD/YY HH:mm:SS VM_NAME: Perform incremental backup.
MM/DD/YY HH:mm:SS VM_NAME: Using backup mode 'VSS'.
MM/DD/YY HH:mm:SS VM_NAME: Unable to start backup on vProxy 'VPROXY_NAME': libCURL: function "curl_easy_perform" returned error 60: "SSL certificate problem: invalid CA certificate" .
MM/DD/YY HH:mm:SS Summary of VMs: after iteration 1
MM/DD/YY HH:mm:SS 0 canceled backups
MM/DD/YY HH:mm:SS 1 failed backups
MM/DD/YY HH:mm:SS 0 VMs not in inventory
MM/DD/YY HH:mm:SS 0 successful backups
MM/DD/YY HH:mm:SS 0 waiting backups
MM/DD/YY HH:mm:SS 0 running backups
MM/DD/YY HH:mm:SS 0 savesets output to next action
The action logs are on the NetWorker server:
- Linux:
/nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME - Windows (Default):
C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME - NetWorker: How to use nsr_render_log to render .raw log files
This failure is occurring on the NetWorker server, there are no logs on the vProxy that correspond with the above failure. The issue occurs before a session is established with the vProxy.
There are no communication issues between the vProxy and NetWorker server, both hosts can resolve each other and connect using port 9090: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
Cause
This issue was observed in an environment where the NetWorker server had not been rebooted in many months. The suspected cause is stale certificates used between the NetWorker server and vProxy.
Resolution
Reset the certificates used by the NetWorker server.
- From an elevated prompt on the NetWorker server, stop NetWorker services:
Linux: nsr_shutdown or systemctl stop networker
Windows: net stop nsrexecd /y
- Move the following files to another directory on the host:
Linux:
/nsr/sec/authcerts/SERVER-NAME_9090/nsr/sec/authcerts/SERVER-NAME.cacert
Windows (Default):
C:\Program Files\EMC NetWorker\nsr\sec\authcerts\SERVER-NAME_9090C:\Program Files\EMC NetWorker\nsr\sec\authcerts\SERVER-NAME.cacert
- Reboot the NetWorker server.
- After the host has been rebooted, run the following command from an elevated prompt:
nsrauthtrust -H NETWORKER_SERVER_NAME -P 9090
This command re-creates the /nsr/sec/authcerts/SERVER-NAME_9090 file.
- Restart VM backups.
If backup failures persist, it may be necessary to reregister the vProxies to reset the certificate on the vProxy host: NVP vProxy: How To Unregister/Re-Register a vProxy Appliance
Reregistering the vProxy adds the /nsr/sec/authcerts/SERVER-NAME.cacert back to the NetWorker server.