PowerFlex: Top Dial Home Events and Alert with Recommended Action

Top Dial Home PowerFlex Alerts

Alert codes are subject to change. This is not a full list of all supported event codes. The full list of PowerFlex Core Software Event Codes can be found in the official documentation.

• Dell PowerFlex 4.5.x and PowerFlex Manager 4.6.x Alerts and Events Reference Guide

CONFIGURED_SPARE_CAPACITY_SMALLER_THAN_LARGEST_FAULT_UNIT    

SIO02.04.0000008
Configured spare capacity is smaller than the size of the largest Fault Set    
CONFIGURED_SPARE_CAPACITY_SMALLER_THAN_LARGEST_FAULT_UNIT    
MDM.Storage_Pool.Configured_Spare_Capacity_Smaller_Than_Largest_Fault_Unit    
(Warning)    
Recommended Action: Increase the "spare percentage", configured in the Storage Pool and reserved for failure recovery, so that it is larger than the largest Fault Unit in the Storage Pool.

Related Articles

• PowerFlex 3.X: Configured spare capacity is smaller than the size of the largest Fault Set
• PowerFlex 4.X: False Alerts for Spare Capacity Storage Pool Unbalanced SPARE_CAPACITY_SMALLER_THAN_LARGEST_FAULT_UNIT

SIO01.03.0000001    
Oscillating failures reported    
OBJECT_HAS_OSCILLATING_FAILURES    
System.Oscillating_Failures.Object_has_oscillating_failures SIO01.03.0000001
Recommended Action: Check oscillating failures of the component and take action accordingly. If the oscillating failure does not indicate a problem, change the settings of the oscillating failure window to suppress this alert

SIO01.04.0000001 
No valid MDM credentials are configured in VxFlex OS Gateway    
GW_CONFIGURATION_INVALID_MDM_CREDENTIALS    
System.Credentials.GW_CONFIGURATION_INVALID_MDM_CREDENTIALS    SIO01.04.0000001     (Critical)    
Recommended Action: Configure the MDM credentials in the Powerflex OS Gateway using the FOSGWTool.

SIO01.04.0000002
MDM credentials are not configured in the Powerflex OS Gateway    
MDM_CREDENTIALS_ARE_NOT_CONFIGURED    
System.Credentials.MDM_CREDENTIALS_ARE_NOT_CONFIGURED    
(Critical)    
Recommended Action: Configure MDM credentials on the PowerFlex OS Gateway using FOSGWTool.  Check the Dell EMC PowerFlex Rack Administration Guide for more instruction.

SIO01.07.0000004
All SDCs are disconnected from one SDS    
ALL_SDC_DISCONNECTED_FROM_ONE_SDS    
System.SDC.All_Sdc_Disconnected_From_One_Sds    
(Critical)    
Recommended Action: Check the MDM event log to determine if the SDS decoupled.  Check the SDS health (scli --query_all_sds), generate a full get_info bundle from the Powerflex OS GW, and contact Technical support.
See article PowerFlex: Dial home alert for disconnections between SDC and SDS

SIO01.07.0000005
All SDCs are disconnected from one SDS IP address    
ALL_SDC_DISCONNECTED_FROM_ONE_SDS_IP    
System.SDC.All_Sdc_Disconnected_From_One_Sds_Ip
(Warning)
Recommended Action: Check all nics and switchports on the host running the SDS application. If 1 of the Powerflex OS links is down.  This error will be triggered.  If you are unable to resolve the problem.  Contact Technical Support.
See article PowerFlex: Dial home alert for disconnections between SDC and SDS

SIO01.07.0000007
Disconnected network links between SDCs and SDSs.    
SDC_MULTIPLE_DISCONNECTIONS_FROM_SDS    
System.SDC.Sdc_Multiple_Disconnections_From_Sds    
(Critical)    
Recommended Action: Check the network links between all SDCs and SDSs.  Review switch logs for any config changes.  When multiple SDC's disconnect from a single SDS.  That is a sign of a loss of connectivity or a problem with one SDS.

SIO02.01.0000007
The MDM cluster is degraded, and data is not protected    
CLUSTER_DEGRADED    
MDM.MDM_Cluster.CLUSTER_DEGRADED    
(Critical)    
Recommended Action: Check that all MDM cluster nodes are functioning correctly, and fix and replace faulty nodes, if necessary, in order to return to full protection.
See aarticle PowerFlex: SDC_MULTIPLE_DISCONNECTIONS_FROM_SDS - SIO01.07.0000007 or SDC_CANNOT_CONNECT_TO_SOME_SDS -SIO01.07.0000010

SIO02.01.0000009
The MDM is not operating in Clustered mode    
MDM_NOT_CLUSTERED_VOLUMES_EXIST    
MDM.MDM_Cluster.MDM_Not_Clustered_Volume_Exist    
(Critical)    
Recommended Action: The MDM cluster was manually set to SINGLE mode. Working in SINGLE mode is not recommended. Single mode means that there is only one copy of the MDM repository. If you lose this copy, all System configurations and all the data on all the existing volumes will be lost. Please verify that this is an expected operation. Prepare the cluster modules (if needed), and return to CLUSTERED mode as soon as possible.

SIO02.04.0000001
Storage Pool has failed capacity    
STORAGE_POOL_HAS_FAILED_CAPACITY    
MDM.Storage_Pool.Storage_Pool_has_Failed_Capacity    
(Critical)    
Recommended Action: For the given Storage Pool, for some blocks, both primary and secondary copies are inaccessible. Check and fix the state of all devices in the Storage Pool and all the server's holding devices in the Storage Pool.

SIO02.04.0000002
Storage Pool has degraded capacity    
STORAGE_POOL_HAS_DEGRADED_CAPACITY    
MDM.Storage_Pool.Storage_Pool_has_Degraded_Capacity    
(Error)    
Recommended Action: For the given Storage Pool, for some blocks, one of the two copies is inaccessble. Check if a server is offline or if there is another server hardware-related issue. Check if a storage device is down.

SIO02.04.0000003    
Capacity utilization above critical threshold    
CAPACITY_UTILIZATION_ABOVE_CRITICAL_THRESHOLD    
MDM.Storage_Pool.Capacity_Utilization_Above_Critical_Threshold    
(Critical)    
Recommended Action: The capacity utilization of the Storage Pool is reaching a critical threshold. Remove unneeded volumes and snapshots, if possible, or add physical storage.
See article PowerFlex: CAPACITY_UTILIZATION_ABOVE_CRITICAL_THRESHOLD  SIO02.04.1000003 or SIO02.04.0000004

SIO02.04.0000004    
Capacity utilization above high threshold    
CAPACITY_UTILIZATION_ABOVE_HIGH_THRESHOLD    
MDM.Storage_Pool.Capacity_Utilization_Above_High_Threshold    
(Error)    
Recommended Action: The capacity utilization of the Storage Pool is reaching a high threshold. Remove unneeded volumes and snapshots, if possible, or add physical storage.
See article PowerFlex: CAPACITY_UTILIZATION_ABOVE_CRITICAL_THRESHOLD  SIO02.04.1000003 or SIO02.04.0000004

SIO02.04.0000005    
Failure recovery capacity is below the threshold    
FAILURE_RECOVERY_CAPACITY_BELOW_THRESHOLD    
MDM.Storage_Pool.Failure_Recovery_Capacity_Below_Threshold    
(Error)    
Recommended Action: The capacity available for recovery in a degraded storage event is lower than the predefined threshold. Replace failed hardware or add more physical storage.
See article PowerFlex: Dial home alert SIO02.04.0000005 - FAILURE_RECOVERY_CAPACITY_BELOW_THRESHOLD

SIO02.04.0000008
Configured spare capacity is smaller than the size of the largest Fault Set    
CONFIGURED_SPARE_CAPACITY_SMALLER_THAN_LARGEST_FAULT_UNIT    
MDM.Storage_Pool.Configured_Spare_Capacity_Smaller_Than_Largest_Fault_Unit    
(Warning)    
Recommended Action: Increase the "spare percentage", configured in the Storage Pool and reserved for failure recovery, so that it is larger than the largest Fault Unit in the Storage Pool.

SIO02.04.0000009
The Storage Pool relies too heavily (over 50%) on capacity from a single SDS or Fault Set. 
Balance capacity over other SDSs or Fault Sets.    
STORAGE_POOL_UNBALANCED    MDM.Storage_Pool.STORAGE_POOL_UNBALANCED    
(Error)    
Recommended Action: Move some physical disks from the large SDS to the others, or add disks to the smaller SDS in order to approximate the capacity of the large SDS as much as possible.
See article PowerFlex: STORAGE_POOL_UNBALANCED SIO02.04.0000009

SIO03.02.0000002
Device test is done and device is pending activation    
DEVICE_PENDING_ACTIVATION    
SDS.Device.Device_Pending_Activation    
(Warning)    
Recommended Action: The SDS device has been added and tested. Activate the SDS device.

SIO03.02.0000003    
Device has fixed read errors    
FIXED_READ_ERROR_COUNT_ABOVE_WARNING_THRESHOLD    
SDS.Device.FIXED_READ_ERROR_COUNT_ABOVE_WARNING_THRESHOLD    
(Error) 
Recommended Action: if counter > 0    Read from the SDS device failed. Data was corrected from the other copy. No action is required, but note that the device might be faulty.

SIO03.02.0000005
Device failed: All IO to the device will be stopped, and data will be relocated to another device.    
DEVICE_ERROR_ERROR    
SDS.Device.Device_ErrorError    
(Critical)    
Recommended Action: Check the device, and if necessary, open a case to have it replaced.
See article PowerFlex: SIO03.02.0000005 Device failed: All I/O to the device will be stopped

SIO03.02.0000013
The disk may be about to fail, or may be operating with reduced performance.    
SMART_AGGREGATED_STATE_FAILED_NOW    
SDS.Device.SMART_Aggregated_State_Failed_Now    
(Error)    
Recommended Action: Sanitize NVDIMM See article PowerFlex 4.X: SMART_AGGREGATED_STATE_FAILED - DAX Device - SIO03.02.0000013

SIO06.01.0000001
Errors have been detected on the Physical Disk    
PHYSICAL_DRIVE_BAD_STATE    
Storage_Controller.Physical_Drive.BAD_STATE    
(Critical)    
Recommended Action: Replace the Physical Disk.
See article PowerFlex: Dial home alert SIO06.01.0000001 - PHYSICAL_DRIVE_BAD_STATE

SIO07.02.0000001
The storage controller is not operating optimally    
STORAGE_CONTROLLER_INVALID_STATE    
Node.Storage_Controller.Invalid_State    
(Critical)    
Recommended Action: The storage controller may be faulty, and should be replaced.

SIO07.03.0000007
CPU has invalid voltage state state    
CPU_SOCKET_INVALID_VOLTAGE    
Node.Cpu_Socket.Invalid_Voltage
(Critical)    
Recommended Action: Check the chassis power supply

SIO07.04.0000007    
Replace DIMM    
DIMM_IN_ERROR_STATE    
Node.Ram.Error_State     
(Error)    
Recommended Action: Replace DIMM

SIO07.04.0000008
DIMM reports Degraded state    
DIMM_IN_DEGRADED_STATE    
Node.Ram.Degraded_State
(Warning)    
Recommended Action: Replace DIMM as soon as possible

SIO07.05.0000005
SDC is not installed on this node    
NODE_WITH_NO_SDC    
Node.Node.NODE_WITH_NO_SDC    
(Error)    
Recommended Action: Consider installing an SDC on this node, so that it can use Powerflex OS volumes

SIO07.06.0000001
Drive is ready to be removed    
PHYSICAL_DRIVE_REMOVED_FROM_OS    
Node.Physical_Drive.Ready_For_Removal        
(Warning)    
Recommended Action: The drive is ready to be removed from the node.

SIO09.02.0000002
Power supply is missing    PSU_NOT_AVAILABLE    
Chassis.Psu.Not_Available    
(Error)    
Recommended Action: Install a new Power Supply Unit, or if there is an existing PSU, verify that it is properly connected.

SIO11.01.0000001
Network Adapter Link is down    
NIC_PORT_DOWN    
Nic.Nic_Port.Nic_Port_Down    
(Critical)    
Recommended Action: The network link is down in the network adapter. Verify that the network adapter is enabled in the operating system, and verify that the network cable is connected properly at both ends. Verify the switch port is not in errdisable mode. Perform a shut and no shut on the port that is down.  If the issue is not resolved.  Please contact technical support.

SIO12.01.0000003
The automatic log collection directory is full    
AUTOMATIC_LOGS_COLLECT_DIRECTORY_ABOVE_HIGH_THRESHOLD    
System.Automatic_Collect_Logs.REACHED_CAPACITY_LIMIT    
(Warning)    
Recommended Action: Delete some files from the directory: (Linux) /opt/emc/scaleio/gateway/temp/scaleio-auto-collect-logs/ or (Windows) C:\Program Files\EMC\ScaleIO\Gateway\Temp\scaleio-auto-collect-logs\

Top Dial Home generated Hardware Alerts

CPU0704    
CPU arg1 machine check error detected.   
Recommended Action:  
1) Check system and operating system logs for exceptions. If no exceptions are found continue. 
2) Turn system off and remove input power for one minute. Re-apply input power and turn system on. 
3) Make sure the processor is seated correctly. 
See article PowerEdge: CPU0704 Machine Check Errors Reflect Inaccurate CPU Index

FAN0001     
Fan arg1 RPM is less than the lower critical threshold.     
Recommended Action: Remove and re-install the fan. If the issue persists, replace the fan.
See article PowerEdge: Troubleshooting FAN0000 and FAN0001 Errors

MEM0001  
Multi-bit memory errors detected on memory device at location(s) DIMM_XX.   
MEM0001 results in the server rebooting due to the fatal error. Memory retraining will automatically occur during that boot.  
Recommended Action: With either of these correctable or uncorrectable (multibit) memory errors, the resulting memory retraining on reboot/restart may "self-heal" the failing DIMM by optimizing the signal timing/margining for each DIMM/slot. A DIMM replacement for these errors is not necessary unless memory retraining fails (UEFI0106) during boot or these same errors continue to occur.
See article PowerEdge: What is DDR4 Self-healing with Intel Xeon Scalable Processors

MEM0702    
Correctable memory error rate exceeded for DIMM_XX.  
Initial Steps: If MEM0702 is logged in the VC events/ Dial home/SEL/LifeCycle log. Results in Post Package Repair being scheduled for the next reboot (warm or cold).  After the reboot, verify that the PPR operation was successfully performed. See "Note":
See article PowerEdge: DDR4 Self-healing on Dell PowerEdge Servers with AMD Rome and Milan processors

MEM9060 
"The PostPackage Repair operation is successfully completed on the Dual In-line Memory Module (DIMM) device that was failing earlier."
A DIMM replacement for these correctable memory errors is not necessary unless the PPR operation fails after the reboot. An example of a failing PPR message is:
Critical - Message ID UEFI0278 - "Unable to complete the Post Package Repair (PPR) operation because of an issue in the DIMM memory slot X."

Recommended Action:  If the issue is not resolved during PPR (See note above).  Replace the faulty dimm.
See article PowerEdge: DDR4 Self-healing on Dell PowerEdge Servers with AMD Rome and Milan processors

MEM8000    
Correctable memory error logging disabled for a memory device at location.
Note: In a situation where you encounter message ID MEM8000 (Correctable memory error logging disabled for a memory device at location DIMM_XX) which appears in isolation (ie – not in a similar time-frame) to any corresponding MEM0005/MEM0701/MEM0702 messages, it will not result in a PPR being scheduled for the next reboot.

Recommended Action: MEM8000 in isolation or with a corresponding CPU MCE (machine check exception) is an indication of a general failure of the DIMM module and is not a situation where the correctable or uncorrectable buckets will initially overflow. This type of memory event should be treated as a DIMM failure and the listed DIMM module should be replaced at the customer’s earliest convenience. 
Note: If the MEM8000 alert does not correspond with a CPU MCE.  An upgrade to the bios may resolve the issue without needing to replace the DIMM.
See article PowerEdge: What is DDR4 Self-healing with Intel Xeon Scalable Processors

PDR3  
A solution exists for this issue but intervention from technical support personnel is required. 
Recommended Action: The firmware on the BOSS card may need to be updated. Contact DellEMC Customer Support Center or your service representative for technical assistance.
See article PowerEdge: Understanding Disk Errors: PDR1001, and PDR3 on Physical Drives (PD)

PDR64   
An unrecoverable disk media error occurred on <physical disk> .  
Recommended Action:  Replace faulty disk
See article PowerProtect One DM5510 PDR64 An Unrecoverable Disk Media Error Occurred on -arg1-

PSU0003      
The power input for power supply  is lost.    
Recommended Action: Deassert then assert the PSU, replace if issue is not resolved with a R&R.
See article PowerEdge: How to Troubleshoot a Server Power Supply Unit

TMP0104    
The system board <name> temperature is outside of range.  
Recommended Action:  Check system operating environment and review event log for fan failures.

VDR58    
Bad block medium error is detected at block arg1 on arg2.  
Recommended Action: Replace faulty disk:  Reason:  Recovering a physical disk bad block depends on the RAID level and state of the virtual disk. If a virtual disk is redundant, the controller can recover a bad block on a physical disk. If a virtual disk is not redundant, then the physical disk bad block results in a virtual disk bad block. 

VLT0204     
The system board arg1 voltage is outside of range.     
Recommended Action: 
1) Review system logs for power supply exceptions 
2) Reduce system to minimum configuration, inspect and re-install system cables. 
See article PowerEdge: Understanding the VLT0204 Error Codes

• PFXM03 Appliance is in a critical state with {dir} at 90+ percent full Recommended Action:
  1) Delete unnecessary data
  2) Reclaim Thin Volumes deleted space using article https://www.dell.com/support/kbdoc/000158378
  3) Delete Snapshot
  4) Change Critical Threshold above 90% to allow Rebalance See article https://www.dell.com/support/kbdoc/000025301
  5) Consider expanding Storage Pool by adding more SDSs or adding more disk if SDSs have free slots