NetWorker: How to enable RabbitMQ Debugging

RabbitMQ

NetWorker servers use RabbitMQ as a message Broker agent service. It provides a message queuing mechanism in between the NetWorker server process (nsrd) and other services used by NetWorker, for example: AUTHC, NetWorker Management Console (NMC), NetWorker Web User Interface (NWUI), and so on. RabbitMQ service issues cause the NetWorker server service to shut down; general messages are logged in the NetWorker server's daemon.raw:

• Linux: /nsr/logs/daemon.raw
• Windows: ..\Program Files\EMC NetWorker\nsr\logs\daemon.raw

  137911 YYYY-MM-DD HH:mm:SS AM/PM  5 5 0 313919296 1615791 0 NW_SERVER-NAME nsrctld NSR critical Aborting startup sequence: Process /opt/nsr/rabbitmq-server-3.8.18/bin/rabbitmq-server exited in less than 10 seconds at startup: exit code 0
  127108 YYYY-MM-DD HH:mm:SS AM/PM  5 5 0 313919296 1615791 0 NW_SERVER-NAME nsrctld NSR critical Failed to start all daemons; shutting down...

• How to render the daemon.raw: NetWorker: How to use nsr_render_log to render .raw log files

Log Files:

The log path varies depending on the rabbitmq-server version used. NetWorker versions may include newer versions of RabbitMQ. Version-specific details can be found in the NetWorker Installation and NetWorker Administration guides, available through: https://www.dell.com/support/home/en-us/product-support/product/networker/docs

Debugging:

Debugging options can vary depending on the RabbitMQ version. The rabbitmq-server folder includes the version. For example:

1. NetWorker 19.6-19.9: 3.8.18
2. NetWorker 19.10-19.11: 3.11.16
3. NetWorker 19.12: 3.12.13
4. NetWorker 19.13: 3.12.14

1. Linux: /opt/nsr/rabbitmq-server-#.#.#
2. Windows: ..\Program Files\EMC NetWorker\nsr\rabbitmq-server-#.#.#
   1. Make a copy of the original rabbitmq.config.
      Linux: /opt/nsr/rabbitmq-server-#.#.#/etc/rabbitmq/rabbitmq.config
      Windows: ..\Program Files\EMC NetWorker\nsr\rabbitmq-server-#.#.#\etc\rabbitmq.config
   2. Edit the rabbitmq.config file to contain the following lines:
      Linux:

      %% {log_levels, [{channel, info}, {connection, info}, {federation, info}, {mirroring, info}]},
            {log_levels, [{connection, debug}, {queue, debug}]},
                 {log,
                      [{file, [{level, debug}]},
                      {categories,
                          [{connection,
                              [{level, debug}]
                          },
                          {queue,
                              [{level, debug}]
                          }
                          ]
                      }]
                  },

      NOTE: The comment line (%%) is there already, the changes can be added under this line.
      Windows:
      Add the following highlighted lines at the top of the file.

      [  
       {rabbit, 
       [{tcp_listeners, []}, 
       {log_levels, [{connection, debug}, {queue, debug}]},
                 {log,
                      [{file, [{level, debug}]},
                      {categories,
                          [{connection,
                             [{level, debug}]
                          },
                          {queue,
                              [{level, debug}]
                          }
                          ]
                      }]
                  },            
          {ssl_listeners, [5671]}, 

   3. Attempt to start NetWorker services:
      Linux: systemctl restart networker
      Windows: net stop nsrd /y && net start nsrd
   4. Monitor the rabbit@localhost.log (if Linux), rabbit@NODE.log (if Windows), and crash.log for errors which may be preventing RabbitMQ startup.

When RabbitMQ is running, the NetWorker server should have port 5671 (SSL) listening.

nsrports -t localhost -p 5671

You can use the following commands to get the Process ID (PID) of the process listening on 5671. The process associated with port 5671 should be from rabbitmq-server.

Linux:

netstat -apno | grep 5671
ps -ef | grep PID

Example:

nve:~ # netstat -apno | grep 5671
tcp        0      0 192.168.9.101:52676     192.168.9.101:5671      ESTABLISHED 2607/gstd           keepalive (5922.75/0/0)
tcp        0      0 :::5671                 :::*                    LISTEN      5238/beam.smp       off (0.00/0/0)
tcp        0      0 192.168.9.101:52654     192.168.9.101:5671      ESTABLISHED 2128/java           off (0.00/0/0)
tcp        0      0 ::1:5671                ::1:50974               ESTABLISHED 5238/beam.smp       off (0.00/0/0)
tcp        0      0 ::1:50974               ::1:5671                ESTABLISHED 5519/nsrjobd        keepalive (5922.75/0/0)
tcp        0      0 192.168.9.101:52668     192.168.9.101:5671      ESTABLISHED 2128/java           off (0.00/0/0)
tcp        0      0 192.168.9.101:5671      192.168.9.101:52654     ESTABLISHED 5238/beam.smp       off (0.00/0/0)
tcp        0      0 192.168.9.101:5671      192.168.9.101:52676     ESTABLISHED 5238/beam.smp       off (0.00/0/0)
tcp        0      0 192.168.9.101:5671      192.168.9.101:52668     ESTABLISHED 5238/beam.smp       off (0.00/0/0)

nve:~ # ps -ef | grep 5238
root      5238  5232  0 Jun05 ?        00:02:55 /opt/nsr/rabbitmq-server-3.11.16/erts-13.2.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -B i -- -root /opt/nsr/rabbitmq-server-3.11.16 -bindir /opt/nsr/rabbitmq-server-3.11.16/erts-13.2.2/bin -progname erl -- -home /nsr/rabbitmq -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false
root      5247  5238  0 Jun05 ?        00:00:00 erl_child_setup 1024
root     26598 26408  0 11:42 pts/0    00:00:00 grep --color=auto 5238

Windows:

netstat -ano | findstr 5671
wmic process where "ProcessID='PID'" get ProcessID,ExecutablePath

Example:

PS C:\Users\Administrator.AMER> netstat -ano | findstr 5671
  TCP    0.0.0.0:5671           0.0.0.0:0              LISTENING       356
  TCP    192.168.9.123:5671     192.168.9.123:63097    ESTABLISHED     356
  TCP    192.168.9.123:5671     192.168.9.123:63098    ESTABLISHED     356
  TCP    192.168.9.123:63097    192.168.9.123:5671     ESTABLISHED     6324
  TCP    192.168.9.123:63098    192.168.9.123:5671     ESTABLISHED     6324
  TCP    [::]:5671              [::]:0                 LISTENING       356
  TCP    [::1]:5671             [::1]:63105            ESTABLISHED     356
  TCP    [::1]:5671             [::1]:63106            ESTABLISHED     356
  TCP    [::1]:63105            [::1]:5671             ESTABLISHED     4676
  TCP    [::1]:63106            [::1]:5671             ESTABLISHED     776

PS C:\Users\Administrator.AMER> wmic process where "ProcessID='356'" get ProcessID,ExecutablePath
ExecutablePath                                                         ProcessId
C:\Program Files\EMC NetWorker\nsr\rabbitmq-server-3.8.18\bin\erl.exe  356

NetWorker: How To Enable AUTHC DEBUG for Troubleshooting Purposes.