NVP vProxy: VM Backups fail Unable to start backup on vProxy. Received an HTTP code 500
Summary: NetWorker VMware Protection (NVP) vProxy Virtual Machine (VM) backup workflows fail. The error observed in the workflow log reports: "Unable to start backup on vProxy 'vProxy_Name': Received an HTTP code: 500," ...
Symptoms
NetWorker VMware Protection (NVP) vProxy Virtual Machine (VM) backup workflows fail.
The rendered workflow logs on the NetWorker server report the following errors:
- Linux: /nsr/logs/policy/Policy_Name/Workflow_Name/Action_Name_JobId.raw
- Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\policy\Policy_Name\Workflow_Name\Action_Name_JobId.raw
- NetWorker: How to use nsr_render_log to render .raw log files
YYYY-MM-DD HH:mm:SS 3 5 0 6156 20212 0 NetWorker_Server_Name nsrvproxy_save NSR error VM_Name: Unable to start backup on vProxy 'vProxy_Name': Received an HTTP code: 500, libCURL message: "", vProxy message: "Error received from vProxy ="-500: Unable to send 'StartSession' message to backupEngine. Did not receive a response message with correlation ID '.". ", url: "https://vProxy_Name:9090/api/v1/BackupVmSessions", body: "{JSON_Body}"
The following logs on the vProxy report connection failures with localhost:5672:
- /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-engine.log
- /opt/emc/vproxy/runtime/logs/vrapid/vrapid-engine.log
Cause
The rabbitmq process on the vProxy is faulting due to IPv6 loopback failure on the vProxy OS.
RabbitMQ is used by the vProxy vrapid engine to communicate with the NetWorker server regarding protection operations. rabbitmq startup failures are found in the vProxy's /var/log/messages file.
rabbitmq reports the following in /var/log/rabbitmq/localhost.log.
s2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> BOOT FAILED 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> =========== 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> Error during startup: {error, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {could_not_start_listener,"::1",5672, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {{shutdown, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {failed_to_start_child, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {ranch_embedded_sup, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {acceptor,{0,0,0,0,0,0,0,1},5672}}, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {shutdown, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {failed_to_start_child, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {ranch_listener_sup, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {acceptor,{0,0,0,0,0,0,0,1},5672}}, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {shutdown, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {failed_to_start_child,ranch_acceptors_sup, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {listen_error, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> {acceptor,{0,0,0,0,0,0,0,1},5672}, 2025-10-03 18:24:38.706221+00:00 [error] <0.491.0> eaddrnotavail}}}}}}},
The process is failing to start the IPv6 loopback listener for rabbitmq port 5672. In this instance, the ip a command shows that there is no inet6 loopback address (::1). This resulted in the rabbitmq service failing to start.
All attempts to reconfigure the IPv6 loopback at the OS level failed.
No cause could be determined, no known changes were made to the vProxy; however, the expected default configuration includes IPv6 loopback.
Resolution
Redeploy the vProxy appliance. This ensures that the vProxy is deployed with default configuration parameters and settings. The vProxy appliance does not contain any databases or critical data required for subsequent VMware protection operations to succeed. It is only used to pass data between VMware and Data Domain, and contain historic logs about previous backup or restore jobs.
The vProxy appliance can be redeployed following: NVP vProxy: How To Upgrade or Redeploy the NVP vProxy Appliance
Workaround:
Disable IPv6 listener for rabbitmq on the vProxy.
- Open an SSH session to the vProxy and log in as admin.
- Switch to root:
sudo su - - Modify the /etc/rabbitmq/rabbitmq.config file:
vi /etc/rabbitmq/rabbitmq.config
Change it from:
[
{rabbit,
[{tcp_listeners,
[{"127.0.0.1", 5672}, {"::1", 5672}]}]
}
].
To:
[
{rabbit,
[{tcp_listeners,
[{"127.0.0.1", 5672}]}]
}
].
- Save the changes. Press the ESC key, then enter
:wq!, then use the Enter key. - Reboot the vProxy Virtual Machine (VM).
This workaround should allow for the rabbitmq service to start using IPv4 loopback only; however, it was observed that the logging output did not function correctly. When a backup or restore failure occurs, the logging would not sufficiently detail the failure. The vProxy VM must be redeployed to fix the system level issue preventing IPv6 loopback from working.
Additional Information
In the environment this issue was observed, the redeployed vProxy appliance functioned correctly.