NVP-vProxy: Unable to register vProxy to NetWorker - SSL connect error/low level system api call failure

The NetWorker VMware Protection (NVP) integration is configured with the vProxy appliance. The vProxy encounters an error while trying to be registered to NetWorker through the NetWorker Management Console (NMC).

MM/DD/YYYY HH:MM:SS networkerserver.domain.com nsrdisp_vproxy NSR warning Received an SSL connect error from VPROXY "vproxy.domain.com" for URL "https://vproxy.domain.com:9090/api/v1/VproxyEngines", retrying request.
MM/DD/YYYY HH:MM:SS networkerserver.domain.com nsrdisp_vproxy NSR info libCURL: function "curl_easy_perform" returned error 35: "An error occurred, a low level system api call failure"
MM/DD/YYYY HH:MM:SS networkerserver.domain.com nsrmon NSR critical libCURL: function "curl_easy_perform" returned error 35: "An error occurred, a low level system api call failure"

The NetWorker server fails to complete a TLS handshake with the vProxy over port 9090. Ensure a network firewall or Router Intrusion Prevention or Detection System (IPS/IDS) software is not preventing the packets from reaching the vProxy.

Ensure all ports listed in the Port Requirements section of the NetWorker VMware Integration Guide can communicate properly. See: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations

The NetWorker server must successfully complete a TLS handshake to port 9090 on the vProxy:

Linux NetWorker server:

curl -kv https://VPROXY_ADDRESS:9090

Windows NetWorker server:

curl.exe -kv https://VPROXY_ADDRESS:9090

The vProxy must successfully complete TLS handshake to port 9090 on the NetWorker server:

curl -kv https://NETWORKER-SERVER_ADDRESS:9090

The vProxy must see the certificate from the NetWorker server. This can be checked using openssl

openssl s_client -connect NETWORKER-SERVER_ADDRESS:9090 -showcerts < /dev/null

The following steps can be taken to capture a Wireshark track dump during the vProxy registration.

• Enable vrapid debug on the vProxy: NetWorker VMware Protection (NVP)-vProxy: How to Enable Debug Logging
• Identify the IP address that the vProxy uses to communicate with the NetWorker server.
• Identify the IP address that the NetWorker server uses to communicate with the vProxy.
• For Windows NetWorker Servers, run Wireshark, add the vProxy filter: ip.addr == xxx.xxx.xxx.xxx and start capturing packets.
• For Linux NetWorker servers and vProxy appliance, start a tcpdump:

  sudo tcpdump -i any -s 0 -C 500 -w /tmp/$(hostname)_$(date -I).pcap 'host VPROXY_IP-ADDRESS or host NETWORKER-SERVER_IP-ADDRESS'

  • -i specifies interface, you can use any, or specify a system network interface name, such as eth0.
  • -s 0 specifies a snap length of 65535 (the entire frame is captured).
  • -C 500 option indicates a file size of 500,000,000 bytes.
  • -w option indicates the output file location. The output file shown is automatically generated with the system hostname and YYYY-MM-DD that it was run.
  • VPROXY_IP-ADDRESS specify the IP address the vProxy uses to communicate with the NetWorker server.
  • NETWORKER-SERVER_IP-ADDRESS specify the target IP address on the NetWorker server that the vProxy communicates with.
    • Specifying the source and target IP addresses limits the packet capture to only show traffic on the two addresses specified. 
  • The above syntax is just an example, you can use the tcpdump command as per your discretion.
• On the NetWorker Server, set debug level 9 for the nsrdispd process.
  • Windows prompt or Linux terminal:

    dbgcommand -n nsrdispd Debug=9

• Use nsradmin on the NetWorker server to unregister, and reregister the vProxy: NVP vProxy: How To Unregister/Re-Register a vProxy Appliance?
• Wireshark should now show packets being sent/received from the vProxy IP. Once the packets stop or after a few minutes
  • For Windows NetWorker servers, stop the Wireshark trace on the NetWorker Server.
  • Cancel the "tcpdump" command from the vProxy Appliance, and Linux NetWorker server (if applicable).
• Disable vrapid debug on the vProxy: NetWorker VMware Protection (NVP)-vProxy: How to Enable Debug Logging
• On the NetWorker Server, set debug level 0 for the nsrdispd process.
  • Windows prompt or Linux terminal:

    dbgcommand -n nsrdispd Debug=0

To evaluate the logs, gather the following details:

• The vProxy name, vProxy IP, NetWorker Server name, and NetWorker server IP
• The Wireshark and tcpdump files
• The /opt/emc/vproxy/runtime/logs/vrapid/vrapid-engine.log file from the vProxy Appliance
• The rendered NetWorker server daemon.raw:

  nsr_render_log -S today daemon.raw > daemong_registeringvproxy.log 2>&1