NetWorker: How To Produce AUTHC Java Thread and Heap Dumps For Linux NetWorker Server

Summary: This process is intended for NetWorker backup administrators to enable Java Virtual Machine (JVM) monitoring when Java thread and heap dumps are requested by NetWorker support. This may be useful when doing advanced diagnosis of authentication-related issues. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

Prerequisites:

  • The NetWorker authentication (authc) server is installed on a Linux platform.
  • This process requires restarting NetWorker services, so server downtime must be planned accordingly.
  • The windows box used to monitor the NetWorker server requires having the VisualVM tool installed: https://visualvm.github.io/download.html This hyperlink is taking you to a website outside of Dell Technologies.  
  • Oracle Java Development Kit (JDK) must be installed on the Windows box used to launch VisualVM, NetWorker Runtime Environment (NRE) does not support VisualVM.
NOTE: The Windows system must have network access to the NetWorker server.

Configuring the NetWorker server to show information in VisualVM:

1. Connect to the NetWorker server over SSH and switch to root:

sudo su - 
 
2. Identify the NetWorker server IP address:
ifconfig -a
nslookup NETWORKER_SERVERNAME
 
3. Ensure port 9094 is not being used. 
netstat -apno | grep 9094
NOTE: If the port is being used, choose another port and ensure it is not used by another application.

4. Make a copy of the authservice configuration file: 
cp /opt/nsr/authc-server/bin/authservice /opt/nsr/authc-server/bin/authservice_`date -I`

5. Open the authcservice file with a text editor and add the following lines:

vi /opt/nsr/authc-server/bin/authservice

-Djava.security.properties=${AUTHCCONF}/java.security \
-Dauthc.server.banner.location=${AUTHCCONF} \
-Djava.locale.providers=COMPAT \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9094 \  # <--- Replace this another port if port 9094 is in use by another application
-Dcom.sun.management.jmxremote.rmi.port=9094 \  # <--- Replace this another port if port 9094 is in use by another application
-Djava.rmi.server.hostname=10.xxx.xxx.xxx \  # <--- Replace this with NW server IP address
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-XX:+IgnoreUnrecognizedVMOptions \
org.apache.catalina.startup.Bootstrap

6. Restart NetWorker services:

systemctl restart networker

7. After services come back up, confirm the port set in the authservice file is listening:

netstat -apno | grep 9094

The NetWorker authc server is now configured to show information in VisualVM.

Using VisuaVM to create thread and heap dumps for the NetWorker authc server:

NOTE: Depending on the issue being faced, it may be necessary to generate multiple thread and heap dumps manually within a 15-30 second interval. During slow NetWorker authentication, manually generate thread and heap dumps every X seconds from login start until completion or failure. Restarting NetWorker services can temporarily resolve login performance issues. Wait for the issue to reappear before collecting thread and heap dumps. The thread and heap dumps must be collected while the issue is reproducible.

1. Confirm that the Windows system can connect to port 9094 on the NetWorker server:

Powershell: tnc HOSTNAME -port 9094
using tnc to test port  

2. Launch VisualVM.
3. Right-click Remote and select Add Remote Host:

Use visualvm to add a remote host

4. Enter the NetWorker server hostname, click Advanced Settings and add port 9094.

set the remote port in visualvm

5. Click OK.
6. Right-click the NetWorker server entry and select Add JMX Connection.

Add jmx connection in visualvm

7. In the Connection field, enter the NetWorker server hostname followed by :9094.

connect to the networker server jmx connection

8. Click OK.
9 Double-click the JMX entry in the left menu and select the Threads tab:

monitor the jmx threads from visualvm

To produce thread dumps, click Thread Dump which produces a thread dump under the JMX connect:

create a thread dump in visualvm

To save the thread dump, right-click the dump and click Save As:

save the thread dump file

To produce a head dump, right-click the JMX connection and click Heap Dump:

create a heap dump

You are prompted to save the heap dump under the /tmp directory on the NetWorker server:

save the heap dump

10. When you have finished producing the thread and heap dumps. Manually save and zip (compress) the thread dumps from the VisualVM.

11. Manually zip (compress) the heap dumps on the NetWorker server along with the authc logs:

tar cvzf /tmp/`hostname`_`date -I`.tgz /tmp/heap* /proc/cpuinfo /proc/meminfo /nsr/authc/logs

This command creates a bundle with the NetWorker servers hostname and date in the NetWorker server's /tmp directory. Copy it off with an SCP tool:

NetWorker (Linux): How to copy files to/from a Linux NetWorker server.

Additional Information

NetWorker: How To Enable AUTHC DEBUG for Troubleshooting Purposes

Affected Products

NetWorker

Products

NetWorker Family, NetWorker Series
Article Properties
Article Number: 000224315
Article Type: How To
Last Modified: 29 Apr 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.