NVP vProxy: VM Image Restore fails with "5040 calling system(), returns nonzero"

Summary: A NetWorker VMware Protection (NVP) virtual machine (VM) Image Restore fails with the following error logged in the restore session log "[5040] calling system(), returns nonzero." The Data Domain ddfs.info log reports "out of order read" when reading one of the VMs save set files during the recovery. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • During an NVP virtual machine (VM) image restore, the recovery process fails. The following error is observed in the recover session log:
YYYY-MM-DDTHH:MM:SSZ WARN:   [39] Warning occurred while recovering 'Hard disk #': Warning occured while releasing reader resources: Error closing file 'NW_SERVER/POOL/61/42/LONG_SSID/vm-MOREF-disk-key-#######-flat.vmdk' on DataDomain 'DD_HOSTNAME': [5040] calling system(), returns nonzero
Saved recover sessions can be found on the NetWorker server:
Linux: /nsr/logs/recover
Windows: C:\Program Files\EMC NetWorker\nsr\logs\recover

Recover session logs can also be found on the vProxy used during the recovery:
/opt/emc/vproxy/runtime/logs/recycle/vrecoverd/DATE
  • The vProxy /opt/emc/vproxy/runtime/logs/vrecoverd/vrecoverd-boost.log shows:
YYYY-MM-DDTHH:MM:SSZ WARN:   [13386] [140581123307264] ddcl_nfs_readext_remote_async_cb()6965: err: [5040-nfs readext async rpc error (nfs: Unknown error)]
...
YYYY-MM-DDTHH:MM:SSZ ERROR:  [13386] [140581123307264] ddp_read() failed Offset 3386365706240, BytesToRead 1048576, BytesRead 0 Err: 5040-A previous operation on this file has failed with error code: 5040 and no further APIs can be processed until the file is closed
  • vProxy version deployed is version 4.3.0-54 through 4.3.0-60

Cause

vProxy 4.3.0-54 through 4.3.0-60 use Data Domain (DD) boost library 7.13 which contains a defect resulting in the symptoms reported in this KB. The issue was resolved in DD boost library 8.0.

Resolution

Upgrade the vProxy appliances to 4.4.0-3 (or later). 4.4.0-3 was released using ddboost library 8.0. vProxy 4.4 is supported with NetWorker 19.9 and later.

Workaround:

Disable the Data Domain system parameter OST_DO_READAHEAD

WARNING: Using this workaround, the code path in the defect is not encountered and the restore succeeds if no other major network issues are observed. Disabling this parameter is a performance impact; All read actions are synchronous. Disabling this feature requires se mode access on the Data Domain; it is not advised accessing se mode without Data Domain support involvement.

Engage with Data Domain support to confirm if OST_DO_READAHEAD is enabled and to disable it.

Perform the VM restore with OST_DO_READAHEAD disabled.

Affected Products

Data Domain, NetWorker

Products

NetWorker Family, NetWorker Series
Article Properties
Article Number: 000225869
Article Type: Solution
Last Modified: 14 May 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.