Data Domain: Alert on HA Standby Node - EVT-STORAGE-00020: The Active Tier is Unavailable

The following alert is posted on the standby node:

CRITICAL Storage Tier=Active EVT-STORAGE-00020: The Active tier is unavailable.

The active node shows a 'degraded' status:

# ha status detailed
HA System Status: degraded
Interconnect Status: ok
Primary Heartbeat Status: ok
External LAN Heartbeat Status: ok
Hardware compatibility check: ok
Software Version Check: ok

This condition is observed during disk failures and replacements. It is due to the active node updating RAID superblocks at the same moment as the standby node reading the RAID superblocks for a SCAN command. This can sometimes result in the SCAN on the standby node detecting an inconsistent disk group (DG) when there is no issue. During the update of the RAID superblock, the sequence number is incremented. The RAID superblocks are written to each of the disks in a DG. The RAID superblocks are released when the write is complete, after a few milliseconds. If a SCAN from the standby node happens in this time, the code may detect an incomplete DG because only a few RAID superblocks for that DG have been updated at the time of the read from the standby node.

Apr 27 14:00:26 DD9800-foo kernel: [2511105.137647] (E4)DD_RAID: Attempting SCAN, retry 1
Apr 27 14:00:26 DD9800-foo kernel: [2511105.637354] (E4)DD_RAID: Attempting SCAN, retry 2
Apr 27 14:00:27 DD9800-foo kernel: [2511106.141575] (E4)DD_RAID: Attempting SCAN, retry 3

 

This case was handled by having the RAID code return an error code, and the volume code would retry the SCAN on the standby node. The retries by the volume code are successful, and the correct state of the standby node is found. The issue is that RAID does not return the error code in version 7.7.4 (a good status was returned) and the volume does not attempt any further scans. The tier would appear unavailable, after which the volume code would send the alert on the standby node.

Run the following command on either the standby or active node.

# disk rescan

This command results in resolving this condition and clearing the related alerts.