Data Domain: STORAGE-00009 | RAID Protection in Critical State

The STORAGE-00009 alert fires when two or more disks fail and RAID reconstruction cannot proceed, either because no spare disks are available or because reconstruction has stalled.

• The system generates event code: STORAGE-00009
• The following alert message appears:

  Two or more disks are failed and the RAID protection is in critical state for over a considerable period of time. 
  If an additional disk fails, there might be permanent loss of data. The system will shut down.
  

• A RAID6 disk group operates in a degraded state with two failed disks.
• No spare disks are available to support reconstruction.
• RAID reconstruction stalls or fails to progress.
• The system initiates a graceful shutdown to protect data integrity.

The Data Domain Operating System (DDOS) continuously monitors disk health and RAID status. When two or more disks fail and one of the following conditions exists, DDOS triggers a protective system shutdown:

• No spare disks are available for RAID reconstruction.
• RAID reconstruction fails to progress within the monitored time window.

DDOS performs this shutdown to prevent further degradation and potential permanent data loss that would result from an additional disk failure in the already‑critical RAID group.

Known Software/Firmware Causes of Multiple Disk Failures

If multiple disk failures are occurring due to a software or firmware defect rather than physical disk faults, replacing disks alone will not resolve the issue. Check the following articles to determine whether a known root cause applies to your system:

• Data Domain: Multiple Drives Fail During Failover of HA Systems | CA01 CA02 FW issue -  Applies to DD9400 and DD9900 with 8TB drives running CA01 or CA02 firmware. Fix: Upgrade DDOS to 7.11.x, 7.10.1.0, or 7.7.5.1+ (includes CA04 firmware), or apply MDU.

• Data Domain: Drive Firmware Update Issue on DDOS 7.x | 8.0 | 8.1 | 8.2 | 8.3.0.x -  Applies to all DD systems with external storage running DDOS 7.10, 7.13, 8.0, 8.1, 8.2, or 8.3.0.x. RAID module reference count issue during drive firmware updates causes two devices per Disk Group to fail. Fix: Upgrade DDOS to 7.10.1.70, 7.13.1.30, 8.3.1.0+, or >= 8.4.0.x.

• Data Domain: Disks failing daily after upgrade to DDOS 8.3.0.0 Applies to DD6400 and DM5500 (SP-RAID models) running DDOS 8.3.0.0. Drive firmware upgrade script timeout causes daily single-disk failures. Fix: Upgrade to DDOS >= 8.3.1.x or apply MDU. Note: Failed disks must NOT be replaced.

Proactive measures to prevent this condition:

1. Replace failed disks promptly. Each replaced disk becomes an available spare for future RAID reconstruction.
2. Monitor disk health regularly and do not allow failed disks to accumulate in the system.
3. Maintain adequate spare disk inventory so that automatic reconstruction can begin immediately after a failure.

Reactive steps when STORAGE-00009 fires:

Step 1 - Assess the current RAID protection state:

1. 1. Log in to the Data Domain system CLI.
   2. Run the following commands to evaluate disk health and RAID status:
      • # disk show state
      • # disk show failure-history
   3. Count the number of disks in a **Failed** state and identify which disk groups are affected.
   4. Run the following command to confirm whether the STORAGE-00009 alert is active:
      • # alerts show current
   5. Determine the severity of the condition:

      • Condition	Risk Level	Recommended Action
        STORAGE-00009 alert is active; RAID protection is in critical state	Critical — the system may shut down to prevent data loss	Contact Dell ProSupport immediately and communicate the critical RAID state
        Multiple failures span different disk groups; spare disks are available	High - the system is degraded but retains RAID protection	Contact Dell ProSupport and request expedited disk replacement
        A single disk has failed; spare reconstruction is in progress	Moderate - RAID6 protection remains intact	Follow the standard service request replacement process

Step 2 - Contact Dell ProSupport:

1. 1. Gather the following information before contacting support. This accelerates diagnosis and replacement:

      • INFORMATION NEEDED:	HOW TO OBTAIN:
        System model and service tag	# system show serialno
        Current disk states	# disk show state
        Disk failure history	# disk show failure-history
        Active alerts, including STORAGE-00009 output	# alerts show current
        Drive model and firmware version	# disk show hardware
        Number of pending replacement disks	Your existing service request records
        Support Bundle (may be requested if additional investigation is required)	Data Domain: How To Collect a Data Domain Support Bundle

   2. Contact Dell ProSupport or your contracted Support Provider and clearly communicate the situation, including:
      • The RAID protection state (critical, degraded, or reconstructing).
      • The number of failed disks and affected disk groups.
      • Whether replacement disks are pending delivery.

Step 3 - Take interim protective actions while awaiting replacement disks:

1. 1. Monitor disk health by running `disk show state` at least twice daily to detect any additional failures immediately.
   2. Reduce non-critical workload where possible. Pause non-essential backup schedules and replication jobs to lower I/O load on degraded disk groups.
   3. Do not power-cycle the system** unless Dell ProSupport explicitly directs you to do so.
   4. If an additional disk fails in an already-degraded disk group, contact Dell Support immediately - do not wait for the next business day.

Step 4 - Replace failed disks:

1. 1. Replace failed disks promptly as replacement hardware arrives. Each replaced disk becomes an available spare for RAID reconstruction.
   2. After replacement, verify that RAID reconstruction begins by running:
      • # disk show state
   3. Continue monitoring until reconstruction completes and the STORAGE-00009 alert clears.