Data Domain: Drive Firmware Update Issue on DDOS 7.13 | 8.1| 8.2 | 8.3.0.x

• Two devices per Device Group (DG) unexpectedly enter a failed state
• Attempting to fail a third device on the Head Unit results in a system panic (Total Fail state)
• Excessive kern.info WARN log entries
• Degraded disk group status
• Noticeable performance degradation on the DDR

Affected systems:

• DD systems with external storage running early versions of DDOS 7.13.x.x | 8.1.x.x | 8.2x.x | 8.3.0.x
  • A fix has been backported to >=7.13.1.40
  • Customers running 8.1, 8.2 or 8.3.0.x  should update to >=8.3.1.10

During the drive firmware update process, the RAID command check scan may execute multiple times based on the number of devices in the system. Each execution increases the RAID module's reference count in the Linux kernel. On kernel version 5.5 (used in DDOS 7.13, 8.1, 8.2, and 8.3.0.x), this reference count does not decrement. If the count rolls over to zero, the kernel blocks RAID from accessing internal gendisk structures, causing devices to be marked unreadable and moved to a failed state. Each DG tolerates only two failed devices; a third failure on the Head Unit triggers a system panic.

A permanent fix has been integrated into the following DDOS versions:

• DDOS >=7.13.1.40
• DDOS >=8.3.1.10

Workaround:

• Raise a Service Request with Dell Tech Support and reference this KB article (#000331892) to expedite the resolution.
  • Dell Tech Support:
    • Modify the drive firmware upgrade script to return immediately after execution, minimizing the increase in the RAID module reference count.