NetWorker: How to enable RabbitMQ Debugging

Summary: This KB serves as an overview of the rabbitmq service, associated log files, and additional debugging that can be applied.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

RabbitMQ

NetWorker servers use RabbitMQ as a message Broker agent service. It provides a message queuing mechanism in between the NetWorker server process (nsrd) and other services used by NetWorker, for example: AUTHC, NetWorker Management Console (NMC), NetWorker Web User Interface (NWUI), and so on. RabbitMQ service issues cause the NetWorker server service to shut down; general messages are logged in the NetWorker server's daemon.raw:

  • Linux: /nsr/logs/daemon.raw
  • Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw 
    137911 YYYY-MM-DD HH:mm:SS AM/PM  5 5 0 313919296 1615791 0 NW_SERVER-NAME nsrctld NSR critical Aborting startup sequence: Process /opt/nsr/rabbitmq-server-3.8.18/bin/rabbitmq-server exited in less than 10 seconds at startup: exit code 0
127108 YYYY-MM-DD HH:mm:SS AM/PM  5 5 0 313919296 1615791 0 NW_SERVER-NAME nsrctld NSR critical Failed to start all daemons; shutting down...
  • How to render the daemon.raw: NetWorker: How to use nsr_render_log to render .raw log files

Log Files:

The log path varies depending on the rabbitmq-server version used. NetWorker versions may include newer versions of RabbitMQ. Version-specific details can be found in the NetWorker Installation and NetWorker Administration guides, available through: Support for NetWorker | Manuals & Documents 

Linux Log Locations:
/nsr/logs/daemon.raw NetWorker services log
/opt/nsr/rabbitmq-server*/var/log/rabbitmq/log/crash.log RabbitMQ crash logs, this directory contains crash.log# depending on log rotation.
/opt/nsr/rabbitmq-server*/var/log/rabbitmq/rabbit@localhost.log RabbitMQ general log
/opt/nsr/rabbitmq-server*/var/log/rabbitmq/rabbit@localhost_upgrade.log RabbitMQ upgrade log
/nsr/tmp/rabbitmq/cores/erl_crash.dump Crash stack trace logs

 

Windows Log Locations (Default Install Location)
C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw NetWorker services log
C:\Program Files\EMC NetWorker\nsr\tmp\RabbitMQ\log\log\crash.log RabbitMQ crash logs, this directory contains crash.log# depending on log rotation.
C:\Program Files\EMC NetWorker\nsr\tmp\RabbitMQ\log\rabbit@HOSTNAME.log RabbitMQ general log
C:\Program Files\EMC NetWorker\nsr\tmp\RabbitMQ\log\rabbit@HOSTNAME_upgrade.log RabbitMQ upgrade log
C:\Program Files\EMC NetWorker\nsr\tmp\RabbitMQ\cores\erl_crash.dump Crash stack trace logs

Debugging:

  1. Make a copy of the original rabbitmq.config.
  • Linux: /opt/nsr/rabbitmq-server-#.#.#/etc/rabbitmq/rabbitmq.config
  • Windows (Default): C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-#.#.#\etc\rabbitmq.config
NOTE: The RabbitMQ server version differs across NetWorker versions. The "rabbitmq-server" folder includes the RabbitMQ server version.
  1. Edit the rabbitmq.config file to contain the following lines:
    Linux:
%% {log_levels, [{channel, info}, {connection, info}, {federation, info}, {mirroring, info}]},
      {log_levels, [{connection, debug}, {queue, debug}]},
           {log,
                [{file, [{level, debug}]},
                {categories,
                    [{connection,
                        [{level, debug}]
                    },
                    {queue,
                        [{level, debug}]
                    }
                    ]
                }]
            },
NOTE: The comment line (%%) is there already, the changes can be added under this line.
Windows:

Add the following highlighted lines at the top of the file.

[  
 {rabbit, 
 [{tcp_listeners, []}, 
 {log_levels, [{connection, debug}, {queue, debug}]},
           {log,
                [{file, [{level, debug}]},
                {categories,
                    [{connection,
                       [{level, debug}]
                    },
                    {queue,
                        [{level, debug}]
                    }
                    ]
                }]
            },            
    {ssl_listeners, [5671]}, 
  1. On Linux servers, add trace messaging into the rabbitmq-server binary: vi /opt/nsr/rabbitmq-server-#.#.#/bin/rabbitmq-server
    1. At the top of the file, on the same line as #!/bin/sh append -x. Example: 
      [root@nsr ~]# head /opt/nsr/rabbitmq-server-4.1.1/bin/rabbitmq-server
#! /bin/sh -x
      During NetWorker server startup, RabbitMQ trace messaging is logged directly into the /nsr/logs/daemon.raw. Once debugging is no longer needed, remove -x from this line.
  2. Attempt to start NetWorker services:
  • Linux: systemctl restart networker
  • Windows (PowerShell): net stop nsrd /y ; net start nsrd
  1. Monitor the rabbit@localhost.log (if Linux), rabbit@NODE.log (if Windows), and crash.log for errors which may be preventing RabbitMQ startup.
WARNING: The purpose of this debugging is for active troubleshooting. For example, when RabbitMQ fails to start. It should not be left enabled and unattended on an active NetWorker server. If RabbitMQ is operational, this debugging can consume thousands of port connections to java which can stop responding the NetWorker server and its user interfaces. After troubleshooting, remove debugging by restoring the rabbitmq.config file to its default settings. Alternatively, use the backup created before adding debug lines. Restarting NetWorker services is required for RabbitMQ to reload the configuration file after debugging has been disabled.

When RabbitMQ is running, the NetWorker server should have port 5671 (SSL) listening.

nsrports -t localhost -p 5671

You can use the following commands to get the Process ID (PID) of the process listening on 5671. The process associated with port 5671 should be from rabbitmq-server.

Linux:

netstat -apno | grep 5671

Or:

ss -tanp 'sport = :5671'

Using the PID of the beam.smp process, confirm if it is RabbitMQ:

ps -ef | grep PID

If RabbitMQ is running, there should be a beam.smp process listening on port 5671.
Example:

nve:~ # netstat -apno | grep :5671
tcp        0      0 :::5671                 :::*                    LISTEN      31393/beam.smp      off (0.00/0/0)
tcp        0      0 192.168.0.4:50612       192.168.0.4:5671        ESTABLISHED 13960/java          off (0.00/0/0)
tcp        0      0 ::1:5671                ::1:36464               ESTABLISHED 31393/beam.smp      off (0.00/0/0)
tcp        0      0 192.168.0.4:50596       192.168.0.4:5671        ESTABLISHED 13960/java          off (0.00/0/0)
tcp        0      0 ::1:36464               ::1:5671                ESTABLISHED 31927/nsrjobd       keepalive (6849.54/0/0)
tcp        0      0 ::1:56715               ::1:8695                TIME_WAIT   -                   timewait (0.24/0/0)
tcp        0      0 192.168.0.4:5671        192.168.0.4:50612       ESTABLISHED 31393/beam.smp      off (0.00/0/0)
tcp        0      0 ::1:56711               ::1:9436                TIME_WAIT   -                   timewait (23.09/0/0)
tcp        0      0 ::1:56717               ::1:7937                TIME_WAIT   -                   timewait (21.02/0/0)
tcp        0      0 ::1:56719               ::1:9436                TIME_WAIT   -                   timewait (23.09/0/0)
tcp        0      0 192.168.0.4:5671        192.168.0.4:50596       ESTABLISHED 31393/beam.smp      off (0.00/0/0)

nve:~ # ps -ef | grep 31393
root     12207 15447  0 23:09 pts/0    00:00:00 grep --color=auto 31393
root     31393 31385  1 23:01 ?        00:00:05 /opt/nsr/rabbitmq-server-4.1.1/erts-14.2.5.11/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -pc unicode -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -B i -- -root /opt/nsr/rabbitmq-server-4.1.1 -bindir /opt/nsr/rabbitmq-server-4.1.1/erts-14.2.5.11/bin -progname erl -- -home /nsr/rabbitmq -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false
root     31408 31393  0 23:01 ?        00:00:00 erl_child_setup 1024

Windows (PowerShell):

Get-NetTCPConnection -LocalPort 5671 |
  Select-Object LocalAddress, LocalPort, RemoteAddress, RemotePort, State,
                @{Name='PID';Expression={$_.OwningProcess}},
                @{Name='ExecutablePath';Expression={
                  (Get-CimInstance Win32_Process -Filter "ProcessId = $($_.OwningProcess)" -ErrorAction SilentlyContinue).ExecutablePath
                }}

Example:

PS C:\Users\Administrator.NETWORKER> Get-NetTCPConnection -LocalPort 5671 |
>>   Select-Object LocalAddress, LocalPort, RemoteAddress, RemotePort, State,
>>                 @{Name='PID';Expression={$_.OwningProcess}},
>>                 @{Name='ExecutablePath';Expression={
>>                   (Get-CimInstance Win32_Process -Filter "ProcessId = $($_.OwningProcess)" -ErrorAction SilentlyContinue).ExecutablePath
>>                 }}


LocalAddress   : fe80::d3e7:121a:e342:3332%4
LocalPort      : 5671
RemoteAddress  : fe80::d3e7:121a:e342:3332%4
RemotePort     : 52085
State          : Established
PID            : 7720
ExecutablePath : C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.12.14\bin\erl.exe

LocalAddress   : fe80::d3e7:121a:e342:3332%4
LocalPort      : 5671
RemoteAddress  : fe80::d3e7:121a:e342:3332%4
RemotePort     : 51582
State          : Established
PID            : 7720
ExecutablePath : C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.12.14\bin\erl.exe

LocalAddress   : ::1
LocalPort      : 5671
RemoteAddress  : ::1
RemotePort     : 51813
State          : Established
PID            : 7720
ExecutablePath : C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.12.14\bin\erl.exe

LocalAddress   : ::
LocalPort      : 5671
RemoteAddress  : ::
RemotePort     : 0
State          : Listen
PID            : 7720
ExecutablePath : C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.12.14\bin\erl.exe

LocalAddress   : 0.0.0.0
LocalPort      : 5671
RemoteAddress  : 0.0.0.0
RemotePort     : 0
State          : Listen
PID            : 7720
ExecutablePath : C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.12.14\bin\erl.exe

Windows (Command Prompt)

netstat -ano | findstr 5671
wmic process where "ProcessID='PID'" get ProcessID,ExecutablePath

Example:

PS C:\Users\Administrator.AMER> netstat -ano | findstr 5671
  TCP    0.0.0.0:5671           0.0.0.0:0              LISTENING       356
  TCP    192.168.9.123:5671     192.168.9.123:63097    ESTABLISHED     356
  TCP    192.168.9.123:5671     192.168.9.123:63098    ESTABLISHED     356
  TCP    192.168.9.123:63097    192.168.9.123:5671     ESTABLISHED     6324
  TCP    192.168.9.123:63098    192.168.9.123:5671     ESTABLISHED     6324
  TCP    [::]:5671              [::]:0                 LISTENING       356
  TCP    [::1]:5671             [::1]:63105            ESTABLISHED     356
  TCP    [::1]:5671             [::1]:63106            ESTABLISHED     356
  TCP    [::1]:63105            [::1]:5671             ESTABLISHED     4676
  TCP    [::1]:63106            [::1]:5671             ESTABLISHED     776

PS C:\Users\Administrator.AMER> wmic process where "ProcessID='356'" get ProcessID,ExecutablePath
ExecutablePath                                                         ProcessId
C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.8.18\bin\erl.exe  356

Additional Information

NetWorker: How To Enable AUTHC DEBUG for Troubleshooting Purposes
NetWorker: How to Enable REST API Debugging
NetWorker Management Web UI (NWUI): Triage and Troubleshooting Guide

Affected Products

NetWorker

Products

NetWorker Family, NetWorker Series
Article Properties
Article Number: 000212107
Article Type: How To
Last Modified: 29 Apr 2026
Version:  10
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.