Data Domain Virtual Edition (DDVE) system upgrade fails with filesystem shutdown error

Summary: Data Domain Virtual Edition (DDVE) system upgrade fails with filesystem shutdown error.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

During filesystem shutdown, any errors from the cloud provider will cause the filesystem to abort inflight requests.  This is by design since retrying a single cloud request can take up to 5 minutes and there is a time limit for the filesystem to shutdown.

Following an example of an error in ddfs.inf0 and the subsequent abort of the request: search for "Request interrupted before retrying".  The "Active" cloud unit will be marked unavailable and the filesystem will exit with a unclean shutdown.

04/12 12:06:29.060 (tid 0x1618b430): ERROR: CAL cl_async_channel_postprocess_enqueue:672 - Request op:6 uri:https://ddvestorageaccount.blob.core.windows.net/ddvecontainer//d1/ca70f3a8/0000000000137383/0000000000000000 timeout reached after nretries:0 04/12 12:06:29.060 (tid 0x7f8b05316600): ERROR: CAL cal_libcloud_iodone_cb:2310 - CAL I/O operation:CAL_READ_OP for object:/d1/ca70f3a8/0000000000137383/0000000000000000 returned error:Request interrupted before retrying 04/12 12:06:29.061 (tid 0x7f8b05316600): INFO: CAL cal_process_io_error:1293 - Marking cloud unit:Active as UNAVAILABLE

Cause

During filesystem shutdown, any errors from the cloud provider will cause the filesystem to exit in an unclean state.

Resolution

Re-try system upgrade after confirming file system is enabled and running again. 

# filesys status
The filesystem is enabled and running.

Note: To reduce the chances of hitting this problem, customers should stop all the activities (ingest, restore, replication, GC) on DDVE before the system upgrade is performed. 

Contact Dell EMC Data Domain support if the system continues to report upgrade failures with file system shutdown error.

Affected Products

Data Domain
Article Properties
Article Number: 000185767
Article Type: Solution
Last Modified: 12 Jan 2023
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.