DDOS 5.7: Accessing file with CIFS in MTree the destination for MTree replication causes DDFS panics or restarts or stale file handles after upgrade

Data Domain Restorers (DDRs) can be configured to replicate the contents of individual MTrees  on one system to a remote system via MTree Replication (MREPL). Despite the MTree on the destination is read-only, it can still be used for accessing the replicated files for reading / restoring / reporting or further outgoing replication.

If the destination DDR is running DDOS 5.7 and an MTree used as a destination for MREPL is accessed via CIFS, the DD FS process may experience PANICs/restarts (causing an interruption to running backups/restores/replication) due to a defect in the code, with possible PANIC signatures as shown below:
 

PANIC: fmcl/fm_inode.c: fm_iput_unpin: 468: fm_iput_unpin failed. Dirents still present. PANIC: ddr/fm/fm_server.c: fms_write_segs: 1891: str == NULL PANIC: ddr/fm/fm_dm1_access.c: fm_dm1_access_intern: 203: fms_is_private_file_handle(&attr->fh) && cred != g_nocred_p

In addition the CIFS client connecting to the MTree via CIFS may see 'invalid handle' errors. Note that this issue only affects DDRs running DDOS 5.7 and is not applicable to any other release of DDOS.
 

Also, there is another possible manifestation of the same issue which may occur on DDOS 6.x or later releases even when the described defect is fixed (see below for releases). The original fix failed to identify another code path for the same defect (namely, when the destination FS for the MTree being used to access files through CIFS is 100% full) which can result in the same PANIC string below, and which has since been fixed in some DDOS 6.x releases:

09/12 01:56:30.847 (tid 0x7f557410ec20): ERROR: MSG-INTRNL-00001: PANIC: fmcl/fm_inode.c: fm_iput_unpin: 469: fm_iput_unpin failed. Dirents still present.

DDOS 5.7 and later (which completely changed the underlying SMB/CIFS implementation in DDOS) include new functionality allowing stateful CIFS file handles. This issue is caused due to a defect in how these stateful CIFS file handles interact with MTree replication functionality.

Note that a fix for this issue was included in the following hot fix/patch releases of DDOS (note as of September 2019, DDOS 5.7 is no longer a supported release):

• DDOS 5.7.1.10 and later
• DDOS 5.7.2.0 and later
• DDOS 5.7.32.0 (this release is only applicable to Data Domain Virtual Edition/DDVE virtual appliances)

Customers should at this time upgrade to a supported DDOS 6.x release, as support termination for the DDOS 5.7 family ended on January 31 2019.


For the corner case defect which may be hit when the FS acting as destination for MREPL is 100% full, there is no fix in any DDOS 5.7 version, but there is in the following later (and currently supported) releases:

• DDOS 6.0.2.40 and later
• DDOS 6.1.2.30 and later
• DDOS 6.2.0.20 and later

As in this case (when the troubled DD FS is 100%) it may not be immediately possible to upgrade DDOS due to lack of FS space, a possible workaround may be:

• Disable all ingest and protocols being used, as well as replication
• Run clean (even if the FS is 100% full, it should complete fine) and wait for it to complete
• Now that the FS should not be 100% full anymore, take the time to remove additional backups, check for old snapshots, etc. and run clean again
• When the amount of disk space is sufficient to avoid filling up the FS again, re-enable protocols and resume backups
• Plan the upgrade to a fixed release for a later time

If the FS is down or crashing repeatedly, please contact your contracted support provider so we can help with an alternative workaround.