Data Domain: Drive Firmware Update Issue on DDOS 7.x | 8.0 | 8.1| 8.2 | 8.3.0.x

• Two devices per Device Group (DG) unexpectedly enter a failed state
• Attempting to fail a third device on the Head Unit results in a system panic (Total Fail state)
• Excessive kern.info WARN log entries
• Degraded disk group status
• Noticeable performance degradation on the DD

Affected systems:

• DD systems with external storage running early versions of DDOS 7.10 | 7.13 | 8.0 | 8.1 | 8.2 | 8.3.0.x

During the drive firmware update process, the RAID command check scan may execute multiple times based on the number of devices in the system. Each execution increases the RAID module's reference count in the Linux kernel. On kernel versions 4.4 and 5.4 (used in DDOS 7.7, 7.10, 7.13, 8.0, 8.1, 8.2, and 8.3.0.x), this reference count does not decrement. If the count rolls over to zero, the kernel blocks RAID from accessing internal gendisk structures, causing devices to be marked unreadable and moved to a failed state. Each DG tolerates only two failed devices; a third failure triggers a system panic on the Head Unit (Controller).

A permanent fix has been integrated into the following DDOS versions:

• LTS releases: 
  • 7.10.1.70 ||  7.13.1.30 || 8.3.1.0 (or newer)
• Feature Releases:
  • >= 8.4.0.x

Workaround:

• If upgrade is not possible.
• To be completed by Dell Tech Support:
  • Modify the drive firmware upgrade script to return immediately after execution, minimizing the increase in the RAID module reference count.
• Customers: Raise a Service Request with Dell Tech Support and reference this KB article (#000331892) to expedite the resolution.