NVE: Crashes and Operating System Utilization Troubleshooting Guide 

Before generating any log bundles, ensure that the NVE's root file system and /data01 partitions are not full: df -hT

If the NVE root (/) file system is full, reclaim space by deleting old logs. If /data01 is full, it can be extended with:

• NetWorker Virtual Edition: How to increase /data01 partition size (XFS)

System Resources:

Determine the CPU configuration of the NVE: cat /proc/cpuinfo
Determine the Memory configuration of the NVE: cat /proc/meminfo

Ensure that the NVE is sized appropriately based on the environment needs. See the NetWorker Performance Optimization and Planning and NetWorker Virtual Edition Deployment Guides for more information: https://www.dell.com/support/product-details/product/networker/overview

NVE Operating System Logs:

• /data01/crash - Review the crash directory for any kdump logs the /etc/sysconfig/kdump shows the kdump level as "0" (nonstripped).
• /var/log/messages - This file has all the global system messages that are located inside, including the messages that are logged during system startup.
• Expanded /var/log. You can collect an expanded /var/log/ bundle using the SUSE supportconfig command.

1. Collect the full supportconfig bundle: supportconfig -A
2. This generates a .txz bundle under /var/log. This bundle can be copied off the NVE with an SCP agent.

To generate a bundle containing the above items, run:

tar cvzf /space/`hostname`-supportconfig-`date -I`.tgz /data01/crash /var/log

If SSH/SCP is not working on NVE, see: NetWorker (Linux): How to copy files to/from a Linux NetWorker server.

CPU and Memory Monitoring:

There are various different utilities, one or more can be used to diagnose a system performance issue.

top

top can be used to monitor CPU and memory usage at any given interval and append to a specified log file. 

1. Log in to the NVE appliance as admin, using SSH or console connection.
2. Switch to root: sudo su -
3. In the root user's home directory (/root/), create a script: vi check_cpu_mem.sh

#!/bin/bash
while true; do
        CMD="echo ; echo ; date ; top -b1 -n1 -o VIRT; echo ; ps aux | sort -k3 -rn"
        eval $CMD >> /space/check_cpu_mem.log
        sleep 300 # Interval (in seconds) at which script will repeat
done

4. Make the script executable: chmod +x /root/check_cpu_mem.sh
5. To run the script in the background so that it is not interrupted run: nohup /root/check_cpu_mem.sh &
6. To stop the script, get the script PID: pgrep check_cpu_mem
7. Use the kill command to stop the script: kill -9 PID

When CPU or memory utilization is high, collect the /space/check_cpu_mem.log along with other operating system logs.

This procedure can also be performed on a per-process basis: NetWorker: Troubleshooting High Memory or CPU usage by Process

nsrwatch

The NetWorker server's nsrwatch utility can be used to monitor NetWorker processes:

NetWorker: How to Troubleshoot Service Response Issues in NetWorker using the nsrwatch utility

pmap

pmap can be used to monitor the memory usage of a specific NetWorker process. The following example demonstrates how to monitor memory for the NetWorker server nsrmmdbd process. NetWorker processes are explained in: NetWorker Processes and Ports

1. Log in to the NVE appliance as admin, using SSH or console connection.
2. Switch to root: sudo su -
3. In the root user's home directory (/root/) create a script: vi pmap_nsrmmdbd.sh

The contents of the script are:

#!/bin/bash
PID=`pgrep -x nsrmmdbd`
while true; do
        CMD="echo ; echo ; date; pmap -x $PID"
        eval $CMD >> /space/pmap_$PID.out
        sleep 300  # Interval (in seconds) at which script will repeat
done

4. Make the script executable: chmod +x /root/pmap_nsrmmdbd.sh 
5. To run the script in the background so that it is not interrupted run: nohup /root/pmap_nsrmmdbd.sh &
6. To stop the script, get the script PID: pgrep pmap
7. Use the kill command to stop the script: kill -9 PID

Collect and review the /space/pmap_PID.out file. The columns in the pmap output represent:

• START: Starting address of the memory segment
• SIZE: Size of the segment
• RSS: Resident Set Size (amount of physical memory used)
• PSS: Proportional Set Size (shared memory divided among processes)
• DIRTY: Memory that has been modified
• SWAP: Amount of memory swapped to disk
• SWAPPSS: Proportional swap size
• PERM: Permissions (such as, read, write, execute)
• MAPPING: Type of mapping (for example, file, anonymous)

NetWorker Service Core Dumps:

NetWorker: How To Use pkgcore for generating core dump bundles

NetWorker Server Logs:

NetWorker logs are detailed in: NetWorker: Log Files and Locations
NVE upgrade logs are detailed in: NetWorker Virtual Edition: Upgrade or Install Failure Log Locations and Collection

The NSRGET utility (version 1.8.2 and later) is designed to collect both NetWorker server and NVE-specific logs. NSRGET can be downloaded from: https://central.dell.com/solutions/Networker-Tools

The recommended command syntax for NVE is: nsrget.sh -o:elrf -p:/space
For additional instructions, see: NetWorker: How to Use the NSRGet NetWorker Data Collection Tool

VMware Logs:

• From vSphere web client, select NVE VM and select Monitor tab.
  • Select Performance Overview.
  • Set a Custom Interval around the time of the crash. Note any continuous high utilization of system resources.
  • Under Tasks and Events, review the tasks and events panels for any errors or failures.
• In the Hosts and Clusters view, Select the vCenter Server object that contains the ESXi hosts from which you want to export logs.
  • Either:
    • Right-click the vCenter Server object and select Export System Logs, or
    • Select the ACTIONS link and select Export System Logs
  • Select the ESXi host that the NVE resides on (shown in the VM's Summary tab).
  • Select the Include vCenter Server, and vSphere UI Client logs option.
  • Click the Next button.
  • Select Gather performance data to include performance data information in the log files.