Isilon: How to check if a power supply has failed

Impact on Data and Cluster Operations

A "Loss in power supply redundancy detected" alert means one power supply unit (PSU) in the node has stopped providing power. The node continues operating on its remaining power supply, and data is not at immediate risk at the time of this alert.

OneFS uses FlexProtect — a distributed data protection mechanism based on Reed-Solomon erasure coding — which can sustain multiple simultaneous node or drive failures (depending on configured protection levels). Even if the affected node were to go offline entirely, data remains accessible from other nodes in the cluster.

However, the node has lost power supply redundancy. If the remaining PSU also fails, the node will go offline. For this reason, treat this alert as high urgency and proceed with the diagnostic and replacement steps below promptly.

Recommendation for large clusters: Ensure all nodes maintain full PSU redundancy and that your data center provides battery-backed UPS and/or generator power. See the Dell PowerScale: Considerations and Best Practices for Large Clusters whitepaper on the Dell Technologies Info Hub for additional guidance.

1. Open an SSH connection on any node in the cluster and log in using the "root" account.
2. Run the following commands:
   1. To check the status of power supplies on one node:
      

      # isi_hw_status | grep -i -A2 "Power Supplies"
      
      Power Supplies OK
      Power Supply 1 good
      Power Supply 2 good

   2. To check the status of power supplies on all nodes:
      

      # isi_for_array -s isi_hw_status | grep -i -A2 "Power Supplies"
      
      RTPIsilon02-1: Power Supplies OK
      RTPIsilon02-1: Power Supply 1 good
      RTPIsilon02-1: Power Supply 2 good
      --
      RTPIsilon02-2: Power Supplies OK
      RTPIsilon02-2: Power Supply 1 good
      RTPIsilon02-2: Power Supply 2 good

   3. The above output shows that all power supplies are OK, so no action is required.
3. If all power supplies are OK, clear the false alerts and monitor the cluster for 24 hours.
   1. To clear only the specific power supply alert, get the group ID and use the 'isi event groups modify' command:
      

      # isi event groups list
      
      ID   Started     Ended       Causes Short                  Lnn  Events  Severity
      -----------------------------------------------------------------------------------
      3074 01/01 00:25 01/01 00:25 POWER Supply Failed           4    1      Critical
      
      # isi event groups modify --resolved=yes --ignore=yes --id=<Group ID from above command>
      
      Example:
      # isi event groups modify --resolved=yes --ignore=yes --id=3074

   2. To clear all old alerts on the entire cluster:
      1. For OneFS 7.x.x.x:
         

         # isi events cancel all
         # isi events quiet all

      2. For OneFS 8.x.x.x:
         

         # isi event bulk --ignore=true
         # isi event bulk --resolved=true

4. If a power supply is failed in the command output, contact Isilon Support for a replacement.

Consolidated Workflow: PSU Redundancy Loss Alert

Follow these steps when you receive a "Loss in power supply redundancy detected" or similar PSU alert:

Table 1. Redundancy Loss Alerts

Watch this video: