NetWorker: Data Unavailable Due to RabbitMQ Issues on Windows Backup Server

Summary: Data Unavailable due to rabbitMQ issues on Backup server.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms

  • The NetWorker server software is installed on a Microsoft Windows server.
  • After a recent upgrade of the NetWorker server, services start; however, nsrwatch and the NetWorker Management Console (NMC) fail to show any backups and manually starting backups fail.
  • The NMC alerts window and NMC gstd.raw report:
Linux: /opt/lgtonmc/logs/gstd.raw
Windows: C:\Program Files\EMC NetWorker\Management\GST\logs\gstd.raw

ERROR generated: "Message bus unable to open socket connection to host 'BACKUP-SRV' on port 5672: a socket error occurred" in file "D:/views/nw/19.8/nsrwebui/modules/nsm/gstnsm.c" line #11364

NetWorker: How to use nsr_render_log

  • Port 5672 or 5671 is not listening on the NetWorker server:
nsrports -t localhost -p PORT
netstat -ano | findstr PORT

NOTE: Port 5672 is the legacy non-SSL port. Port 5671 is used for SSL. In NetWorker 19.7.x and later, port 5672 is disabled and rabbitmq traffic is sent over port 5671.
  • Ran the command to start the rabbitmq server and this was the output:
Configuring logger redirection
Logger - error: {removed_failing_handler,rabbit_log}
13:46:52.589 [error] Error in process <0.206.0> on node rabbit_prelaunch_3804@localhost with exit value:
{eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]}
13:46:52.589 [error] Supervisor inet_gethost_native_sup had child at module inet_gethost_native at <0.206.0> exit with reason {eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]} in context child_terminated
13:46:52.589 [error] gen_server inet_gethost_native_sup terminated with reason: {eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]}
13:46:52.589 [error] CRASH REPORT Process inet_gethost_native_sup with 0 neighbours exited with reason: {eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]} in gen_server:handle_common_reply/8 line 805
13:46:52.589 [error] Supervisor kernel_safe_sup had child inet_gethost_native_sup started with {inet_gethost_native,start_link,undefined} at <0.205.0> exit with reason {eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]} in context child_terminated
13:46:52.605 [error] Error in process <0.206.0> on node rabbit_prelaunch_3804@localhost with exit value:
{eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2258}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]}
13:46:52.636 [error] 

13:46:52.636 [error] BOOT FAILED
BOOT FAILED
13:46:52.636 [error] ===========
===========
13:46:52.636 [error] ERROR: epmd error for host BACKUP-SRV: {could_not_start_server,inet_gethost_native} (unknown POSIX error)
ERROR: epmd error for host BACKUP-SRV: {could_not_start_server,inet_gethost_native} (unknown POSIX error)
13:46:52.636 [error] 

13:46:53.647 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"BACKUP-SRV",{could_not_start_server,inet_gethost_native}} in context start_error
13:46:53.647 [error] CRASH REPORT Process <0.151.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"BACKUP-SRV",{could_not_start_server,inet_gethost_native}}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,\"BACKUP-SRV\",{could_not_start_server,inet_gethost_native}}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"BACKUP-SRV",{could_not_start_server,inet_gethost

Crash dump is being written to: erl_crash.dump...done

Cause

This issue can occur when a rogue file or directory named 'program' exists on the root of the volume where NetWorker is installed. This file or directory may be hidden.

Resolution

Identify if a file or directory named 'program' exists on the root of any volume. If so, delete or rename the file or directory:
  1. Open a CMD prompt using Run as administrator.
  2. Go to the root of the volume.
Example:
cd D:\
  1. Run the following command to show objects, including hidden objects.
dir /a
  1. If a 'Program' is found, delete or rename the object.
  2. Repeat steps 1 - 4 for any other volumes or drive letters.
  3. Restart service
net stop nsrexecd /y
net start nsrd
If NMC is installed on the same system:
net start gstd
  1. Reboot the host at the next opportunity.

Affected Products

NetWorker Family, NetWorker, NetWorker Management Console

Products

NetWorker Management Console
Article Properties
Article Number: 000220828
Article Type: Solution
Last Modified: 09 Oct 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.