NetWorker VMware Protection-vProxy: How to Enable Debug Logging

Watch this Video: How to Enable vProxy Debug Logging.

Usually, it is sufficient to analyze the policy logs of the NetWorker server in /nsr/logs/policy/[policy name], but occasionally it may be necessary to analyze the logs from the vProxy appliance itself and furthermore, to increase the logging level to debug in order to get a clearer idea of the reason for a failure. This is typically only a requirement once NetWorker support has been engaged and the support engineer has requested to debug set to troubleshoot as specific process. 

NVP-vProxy log files on the vProxy appliance are in the following directory /opt/emc/vproxy/runtime/logs.

There is a subdirectory for each of the principal processes which are used for NetWorker operations on the vProxy appliance:

vbackupd

vflrd

vrapid

vrecoverd

For example, the vbackupd subdirectory contains the following logs:

[session-uuid].log             Processing details for a session
vbackupd-engine.log            Requests and problem events including error and panic messages from the session log.
vbackupd-boost.log             DD Boost logging.
vbackupd-snapmgr.log           Snapshot logging
vbackupd-vddk.log              VDDK logging -  VMware Virtual Disk Development Kit

Additional information about vProxy logging and collection is found in the article NVP-vProxy Triage Guide: NetWorker VMware Protection vProxy Triage and Log Overview

Applying Service Debug Method 1:

This method applies to vProxy's version 4.3.x and later. It can be used to enable debug on vProxy services (vrapid, vrecoverd, vbackupd, vflrd). More debugging can be enabled for DD Boost traffic, Virtual Disk Development Kit (VDDK), and snapshot messaging (where applicable).

1. Connect to the vProxy over SSH using the vProxy admin account.
2. Switch to root:

sudo su -

3. Change to the vProxy agent bin directory:

cd /opt/emc/vproxy/bin

4. For the service requiring debug, run the following commands:

Backup Engine: vbackupd

./vProxy_debug.sh -s -d vbackupd -p program -l debug

/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-engine.log

/opt/emc/vrproxy/runtime/logs/vbackupd/BackupVmSession-<session-id>.log
/opt/emc/vrproxy/runtime/logs/recycle/vbackupd/<DATE>/BackupVmSession-<session-id>.log

./vProxy_debug.sh -s -d vbackupd -p ddboost -l all

/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-boost.log

./vProxy_debug.sh -s -d vbackupd -p vddk -l 4

/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-vddk.log

./vProxy_debug.sh -s -d vbackupd -p snapshotmgr -l debug

/opt/emc/vrproxy/runtime/logs/vbackupd/vbackupd-snapmgr.log

Recover Engine: vrecoverd

./vProxy_debug.sh -s -d vrecoverd -p program -l debug

/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-engine.log

/opt/emc/vrproxy/runtime/logs/vrecoverd/RecoverVmSession-<session-id>.log
/opt/emc/vrproxy/runtime/logs/recycle/vrecoverd/<DATE>/RecoverVmSession-<session-id>.log

./vProxy_debug.sh -s -d vrecoverd -p ddboost -l all

/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-boost.log

./vProxy_debug.sh -s -d vrecoverd -p vddk -l 4

/opt/emc/vrproxy/runtime/logs/vrecoverd/vrecoverd-vddk.log

FLR Engine: vflrd

./vProxy_debug.sh -s -d vflrd -p program -l debug

/opt/emc/vrproxy/runtime/logs/vflrd/vflrd-engine.log

/opt/emc/vrproxy/runtime/logs/vflr/FlrMountSession-<session-id>.log
Individual FLR Browse Session Logs: /opt/emc/vrproxy/runtime/logs/vflr/FlrBrowseSession-<session-id>.log
Individual FLR Recover Session Logs: /opt/emc/vrproxy/runtime/logs/vflr/FlrRecoverSession-<session-id>.log

/opt/emc/vrproxy/runtime/logs/recycle/vflrd

./vProxy_debug.sh -s -d vflrd -p ddboost -l all

/opt/emc/vrproxy/runtime/logs/vflrd/vflrd-boost.log

REST Server: vrapid

./vProxy_debug.sh -s -d vrapid -p program -l debug

/opt/emc/vrproxy/runtime/logs/vrapid/vrapid-engine.log

The above commands show enabling debug, these commands can be rerun changing the -l option to reset to default:

Options:

• For Program Log:

• For DD Boost Log:

• For VDDK Log:

• For Snapshot Manager Log:

Applying Service Debug Method 2 (Manually):

1. Log in to the vProxy Appliance as admin over SSH or console and switch to root:

sudo su -

2. Go to the systemd directory:

cd /usr/lib/systemd/system/ 

3. Open the service file that requires debug settings, for example:

vi vbackupd.service 

4. Locate the line with ExecStart= and append with the wanted debugging parameters:

vbackupd, vrecoverd, vflrd, and vrapid engine debug options: { debug | trace | info | warn } (default "trace")
--program-log-level=option

vbackupd, vrecoverd, and vflrd ddboost debug options: { none | error | warn | info | debug | all } (default "trace")
--boost-log-level=option

vbackupd snapshot manager debug options: { warn | info | trace | debug } (default "trace")
--snapmgr-log-level=option

Display version
--version

Multiple options could be used at once, for example if you needed to debug the vbackupd service, ddboost communication, and snapshot manager logging, the vbackupd.service file would include:
ExecStart=/opt/emc/vproxy/bin/vbackupd --program-log-level=debug --boost-log-level=all --snapmgr-log-level=debug

5. Save the changes to the file.
6. Reload the unit configuration file by typing the command:

systemctl daemon-reload

7. Confirm in the NetWorker Management Console that no backups are running and then restart the vbackupd service:

systemctl restart vbackupd.service

The above process is applicable for the services vrapid, vrecoverd, vbackupd, vflrd.

Applying VDDK Debug (Manually):

1. Log in to the vProxy appliance using SSH or console as admin and switch to root:

sudu su -

2. Edit the VixDiskLib.config file to contain the following line:

vi /opt/emc/vproxy/conf/VixDiskLib.config 
vixDiskLib.transport.LogLevel=4

0 = No logging
1 = Errors only
2 = Warnings, and Errors
3 = Important information messages, errors, and warnings
4 = Debug data plus everything else

3. Restart the relevant service:

systemctl restart [service-name].service 

systemctl restart vrecoverd.service

Once the debug logging has been set, the log file associated with the service should contain DEBUG lines.

After completing the debugging process, reset the log level using the same procedure, using the word trace after the equal sign instead of debug. In order for this change to be immediately applicable, a daemon reload and restart is required as outlined above.

DD Boost Precert Debugging:

This process is done with other debugging methods outlined above.

1. Log in to the vProxy appliance using SSH or console as admin and switch to root:

sudu su -

2. Create an empty file called ddboost_precert.ini under /:

touch /ddboost_precert.ini

3. Create the DDLog directory:

mkdir /DDLog

4. Enable DD Boost debug for the service requiring debug:

vbackupd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vbackupd -p ddboost -l debug

vflrd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vflrd -p ddboost -l debug

vrecoverd: /opt/emc/vproxy/bin/vProxy_debug.sh -s -d vrecoverd -p ddboost -l debug

5. Confirm that the ddboost_precert.log is generated:

ls -l /DDLog

​​​​​​

6. When debug operations are no longer required, rerun the following command:

vProxy_debug.sh -l trace

vProxy Selection Logging:

This debugging can be used whenever a VM backup workflow is stuck with the VMs sitting in Waiting to Run. In this scenario, the backup job has started but no vProxy has been picked up to start the backup session. There is no logging of the vProxy selection process in standard logging. We can enable vmbackup_logging on the NetWorker server, which logs the vProxy selection process during the start of VMware Protection Policy. This is useful when there are no communication issues between the NetWorker server and vProxy (Port 9090), and no errors about the vProxy being unavailable in the NMC or daemon.log.

1. Open an Administrative PowerShell prompt (Windows) or a root SSH session (Linux) on the NetWorker server.
2. Create a file without an extension called vmbackup_logging under the NetWorker /nsr/tmp directory:

Linux: 
touch /nsr/tmp/vmbackup_logging

Windows: 
New-Item -Path "C:\Program Files\EMC NetWorker\nsr\tmp\vmbackup_logging" -ItemType File

3. Start the VM backup job.
4. Monitor the rendered daemon.log for Vmproxy Queue and select vproxy messages.

Linux: /nsr/logs/daemon.log
Windows: C:\Program Files\EMC NetWorker\nsr\logs\daemon.log

5. To disable this debugging, delete the vmbackup_logging file from the /nsr/tmp dir.

Related articles:

• NVP-vProxy Triage Guide: NetWorker VMware Protection vProxy Triage and Log Overview
• NetWorker (Linux): How to copy files to/from a Linux NetWorker server.
• NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
• NetWorker: NVP vProxy Troubleshooting VM Backup and Restore Performance and Throughput