NVP-vProxy: VM Loses Connection During Snapshot Removal Process.

Summary: During the snapshot removal step of a vProxy job, the source Virtual Machine (VM) loses connectivity temporarily.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The vProxy backup session log shows when snapshot requests are made:

YYYY/MM/DD HH:MM:SS TRACE: [NW_BUILD] Sending remove snapshot request to Snapshot Manager ...
YYYY/MM/DD HH:MM:SS TRACE: [NW_BUILD] Sending remove snapshot request to Snapshot Manager ...
YYYY/MM/DD HH:MM:SS INFO: [NW_BUILD] Remove snapshot request accepted by Snapshot Manager.
YYYY/MM/DD HH:MM:SS INFO: [NW_BUILD] Remove snapshot request accepted by Snapshot Manager.
NOTE: The session logs can be found on the vProxy under /opt/emc/vproxy/runtime/logs/vbackupd/ or /opt/emc/vproxy/runtime/logs/recycle/vbackupd/


The vProxy vbackupd-snapmgr.log may show an error, for example:

YYYY/MM/DD HH:MM:SS INFO: Found virtual machine "VM-NAME" with MORef "vm-###".
YYYY/MM/DD HH:MM:SS NOTICE: Virtual machine "VM-NAME" requires consolidation of its virtual disks.
YYYY/MM/DD HH:MM:SS ERROR: Failed to complete consolidation of virtual disks for "VM-NAME" (vm-###): Unable to access file since it is locked
YYYY/MM/DD HH:MM:SS WARN: Failed to consolidate virtual disks of virtual machine "VM-NAME" (vm-###): Failed to complete consolidation of virtual disks for "VM-NAME" (vm-###): Unable to access file since it is locked.
YYYY/MM/DD HH:MM:SS WARN: Failed to consolidate virtual disks of virtual machine "VM-NAME" (vm-###): Failed to complete consolidation of virtual disks for "VM-NAME" (vm-###): Unable to access file since it is locked.
YYYY/MM/DD HH:MM:SS NOTICE: Snapshot "snapshot-###" of virtual machine "VM-NAME" (vm-###) was removed.
NOTE: The vbackupd-snapmgr.log can be found under:  /opt/emc/vproxy/runtime/logs/vbackupd/ or /opt/emc/vproxy/runtime/logs/recycle/daemon/


Event logs on the VM operating system may show an error, for example:

The mirroring connection to "TCP://VM-NAME:5022" has timed out for database "out" after 10 seconds without a response. Check the service and network connections"
Event ID: 1479.

The VM's vmware.log (Found in the VMware datastore folder form the VM) may contain additional information about the snapshot removal process.

Cause

NetWorker/vProxy does not remove the snapshot itself. The VMware vCenter controls the creation, management, and deletion of VM snapshots. NetWorker initiates a backup by sending a request to VMware to create a snapshot. At the end, it sends a request to delete the snapshot.

Snapshot removal reduces VM Input/Output Per Second (IOPS). This happens due to extra Virtual Machine File System (VMFS) storage locks from increased metadata updates and added IOP load during the removal process. If target storage runs above 30–40% IOP load, snapshot removal can push it past 80%. This spike often causes latency, harming application performance.

NVP vProxy: Troubleshooting VMware Snapshot Issues during NetWorker backups.

Resolution

Isolation Testing:

To isolate the issue to the specific VMware snapshot removal event, follow the below steps:
 
WARNING: Perform the below steps during a time when connectivity to the VM is not sensitive, for instance during off-peak hours.
  1. Log in to vCenter server or vSphere client.
  2. Create a snapshot on the VM in question.
  3. Leave the snapshot on the VM for a duration of time what a vProxy job runs against the VM.
  4. Initiate a ping command against the VM's IP address/hostname.
Linux: 
ping -D VM_IP
Windows (Powershell):  
ping -t VM_IP | Foreach{"{0} - {1}" -f (Get-Date),$_}
Both of the above commands timestamp the ping. 
  1. Remove the snapshot.
  2. Observe the VM during the snapshot removal.
If while performing the test above the same connectivity issue as during the vProxy backup job are observed, then issue likely exists within the VMware environment. Open a support ticket with VMware support to investigate further.

For VMs stored on an NFSv3 Datastore:

If you are backing up VMs that are stored on an NFSv3 datastore and are observing connection issues, see VMware KB: https://kb.vmware.com/s/article/2010953This hyperlink is taking you to a website outside of Dell Technologies.

VMware reports this specific issue as resolved when using VMware ESXi hosts that are 8.0u2b (or newer)

The following options are available for solutions using NFSv3 for ESXi versions before 8.0u2b. One of the following options must be considered.
  • Deploy an NVP vProxy appliance on the same ESXi host as the VMs that are being stunned.
  • Consider using the NFSv4 protocol for mounting NFS datastores. 
NOTE: Before implementing NFS 4.1, see NFS Protocols and vSphere Solutions - NFS Datastore Concepts and Operations in vSphere Environment.This hyperlink is taking you to a website outside of Dell Technologies.NFS 4.1 uses different locking mechanisms than NFSv3 - See Locking mechanisms.
  • Configure the vProxy appliances which backup these VMs to use NBD instead of HotAdd. Using hot add wherever possible is preferred as it generally performs better than NBD. High NBD load can negatively impact VMware management networks. This option should only be done if the other options mentioned are not possible. 
vProxies can be configured to backup using NBD only. From the NetWorker Management Console (NMC)-->Devices-->VMware Proxies:
NBD session settings
NOTE: Setting 'hot add' to 0 disables the hot add transport mode. 

Additional Information

Additional Troubleshooting:

  • Check the VM for snapshots while no job is running and remove any that are found.
  • Check for orphaned snapshots on the VM. 
  • Reduce the number of concurrent VM backups that are occurring on that datastore/host. See the NetWorker VMware Integration Guide for performance and scalability recommendations.https://www.dell.com/support/home/product-support/product/networker/docs
  • If the VM's CPU resources spike heavily during Snapshot consolidation, consider increasing the CPU reservation for that VM.
  • Move VM to a host with more available resources.
  • If the VM is the vCenter Server Appliances, see Best practices for vCenter server backup and restore In the NetWorker VMware Integration Guide: https://www.dell.com/support/home/product-support/product/networker/docs


VMware KB 1002836 This hyperlink is taking you to a website outside of Dell Technologies. | A snapshot removal can stop a virtual machine for long time.

Review the VMware vCenter support bundle, ensure to collect the ESXi system logs for the ESXi systems that host VMs which observed this symptom. The ESXi host's \var\run\log\vmkernel.log may show storage latency issues during the snapshot operations:

YYYY-MM-DDTHH:mm:SS cpu10:2098132)WARNING: ScsiDeviceIO: 1498: Device naa.############# performance has deteriorated. I/O latency increased from average value of 1429 microseconds to 29874 microseconds.

The storage vendor must be engaged to investigate further.

Affected Products

NetWorker

Products

NetWorker
Article Properties
Article Number: 000167629
Article Type: Solution
Last Modified: 10 Nov 2025
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.