PowerProtect Data Manager: VADP VM Backup Performance, Many VMs Using NBD When Hot add Is Preferred

Summary: VADP running many backups using NBD when Hot add is expected.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • Even though hot add is preferred, many backups are seen running as Network Block Device (NBD).
  • Many VMs failed with the error "The backup had terminated unexpectedly"
  • Error code: 500 -  due to potential NBD related Network File Copy (NFC) exhaustion on vCenter (Note: vCenter error code 500, indicates an internal server error and can occur for many reasons.) 
  • VM backup log "BackupVmSessions-XXX.log" indicates that the VM was backed up as NBD even though Hot add is available

Cause

VADP Transport Optimization feature:
PowerProtect Data Manager includes a VADP Transport Mode Optimizer enabled by default. The optimizer can be seen in the BackupVMSessionn log. 
2023-12-14T00:00:10.118Z INFO:   [8d6fa9f04b913e75;9a002f8c68361a1e]     "TransportModeRequested": "hotadd:nbd:nbdssl",
2023-12-14T00:00:10.118Z INFO:   [8d6fa9f04b913e75;9a002f8c68361a1e] "TransportOptimizationEnabled": true,
Further on in the BackupVMSessions-XXXXXX.log shows the reasoning and chosen transport mode by the optimizer. 
2023-12-13T23:01:00.220Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Transport mode optimization estimates:
2023-12-13T23:01:00.220Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Name,MORef,Mode,Total (sec),Overhead (sec),Read (sec),Read (b/s), Read (M/s),Bytes To Move
2023-12-13T23:01:00.221Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Mode estimate: WINDOWS_VM,vm-31602,hotadd,245,0,245,18662783,4565303296
2023-12-13T23:01:00.221Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Mode estimate: WINDOWS_VM,vm-31602,nbd,+Inf,0,+Inf,0,4565303296
2023-12-13T23:01:00.221Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Mode estimate: WINDOWS_VM,vm-31602,nbdssl,44,0,44,104553178,4565303296
2023-12-13T23:01:00.221Z INFO:   [b75fd74d9f9bf6c0;9fd44f091dfd686e] TransportOptimizer: Best mode is 'nbdssl' in 44 sec (0.00).
The optimizer factors in past backup performance data, the number of disks and the size of each disk to determine the best transport mode for a given backup. It initially relies on some default synthetic performance data for each transport mode until a backup can be taken in that mode for the optimizer to be updated with the real-world performance data.

The optimizer data is lost with a restart of the VM Direct Engine (vProxy) daemon service or if the VM Protection Engine is redeployed.

In order to ensure that the optimizer has real world data to correctly choose between Hot add or NBD for each VM in the vCenter environment, backups using both Hot add and NBD should both be run, preferably several days/weeks for each. Disabling the optimizer and specifically setting the VM Protection Engine to use either Hot add or NBD, for several days each. Once each VM is backed up with each of the modes, the transport optimizer makes the correct prediction of which mode is best for each VM based on real environmental data.

Or if Hot add is the preferred method, and NBD should only be used for those VMs that do not have Hot add access, then disabling the Transport Mode Optimizer is the recommended option.

Resolution

How to disable the VADP Transport optimization - Parameter: TRANSPORT_OPTIMIZATION_ENABLED
1. SSH to the PowerProtect Data Manager appliance and change the directory to below to find the vmdirect.env environmental file. 
    # cd /opt/emc/vmdirect/unit
2. Make a backup copy of the environmental file  
    # cp vmdirect.env vmdirect.env.bak
3. Edit the vmdirect.env environmental file using vi/vim 
    # sudo vi vmdirect.env
Change the TRANSPORT_OPTIMIZATION_ENABLED from true to false

EG
TRANSPORT_OPTIMIZATION_ENABLED=true
to
TRANSPORT_OPTIMIZATION_ENABLED=false
4. Then we must restart the vpod.service for this change to take effect. IMPORTANT: Ensure no VM backup/recover sessions are active. 
# systemctl restart vpod.service.
In the next backup log, BackupVmSessions-XXXX.log (Path to logs on PowerProtect Data Manager appliance -  /opt/emc/vmdirect/runtime/logs/recycle/vpod/), the following line indicates if the Transport Optimization is on or off. 
"TransportOptimizationEnabled": false

Affected Products

PowerProtect Data Manager
Article Properties
Article Number: 000221216
Article Type: Solution
Last Modified: 07 Mar 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.