Data Domain: Alert - IO module has failed | EVT-ENVIRONMENT-00029

This KB article applies to PCIe devices; the same alert can also be generated against a Trusted Platform Module (TPM).

For failed TPM, see KB article Data Domain: Alert: ENVIRONMENT-00029: TPM I/O Module Has Failed

The failure or removal of an I/O card (For example, SAS, NIC, NVRAM, FC) results in loss of communication to the affected card.

• Hot-Swap of PCIe devices is NOT supported. The DD system MUST be powered off to reseat or install a PCIe device
• Hot-swapping triggers an unexpected system reboot.

Alert Example:

Alert Id:       147
Event Id:       EVT-ENVIRONMENT-00029
Event Message:  I/O module has failed
Object:         Enclosure=1:Slot=7
Additional Information: Cause=A hardware fault occurred

A PCIe I/O module (SLIC or PCI card) has either physically failed due to a hardware fault or was removed from its slot while the system was running. 
In some cases, a transient event such as a firmware glitch can trigger a temporary card reset that also generates this alert. 

Step 1 - Confirm the status of the affected card

1. Connect to the DD system through SSH (DDCLI).
2. Run the system show hardware command to display the hardware inventory.
3. Review the output. A failed or removed card appears as (empty) in its slot:  

Example:

sysadmin@DD# system show hardware
Slot   Vendor   Device                            Ports
----   ------   -------------------------------   --------------
M      QLogic   QL41000 4x 10GBase-T (NDC)        Ma, Mb, Mc, Md
1      QLogic   QL41164 4x 10GbE SFP+             1a, 1b, 1c, 1d
2      EMC      NVRAM 16GB Card
3      EMC      PMC Quad Port 12 Gbps SAS         3a, 3b, 3c, 3d
4      Intel    Intel QuickAssist Adapter 8970
5      EMC      Quad Port 16 Gbps Fibre Channel   5a, 5b, 5c, 5d
6      QLogic   QL41000 2x 25GbE                  6a, 6b
7      (empty)         (empty)
8      QLogic   QL41000 2x 25GbE                  8a, 8b
----   ------   -------------------------------   --------------

This example shows slot 7 as (empty) as the card in that slot had failed.

4. If not all slots are populated, compare the current output against a previous Auto-Support report to identify the missing card.

Step 2 - Check whether the alert is transient

This alert can be transient. In some cases, the system successfully recovers the card on its own, for example, after a card reset is triggered by a firmware event.
In most transient examples, the posted alert clears itself and no additional action is required. 

1. Re-run system show hardware after a few minutes.
2. If the card reappears in the output and resumes normal operation, the alert was transient. Clear the alert and monitor the system for recurrence.
3. If the card remains in a failed or (empty) state, proceed to Step 3.

Step 3 - Attempt recovery by rebooting the system

1. Reboot the DD system by running the # system reboot command through SSH.
2. After the system restarts, run system show hardware to verify whether the card has recovered.
3. If the card is detected and functioning normally, clear the alert (# alert clear <alert-id>) and continue to monitor for recurrence.
4. If the card still appears as (empty) or failed, proceed to Step 4.

Step 4 - Reseat or reinstall the card

If the card was removed accidentally, installed into the wrong slot, or needs to be reseated:

1. Power off the DD system by running the # system poweroff command through SSH.
2. Reinstall (or reseat) the I/O card in the correct slot.
3. Power the system back on.
4. Run system show hardware to confirm the card is detected.
5. If the card still fails to appear or function, proceed to Step 5.

Step 5 - Contact Technical Support

If the card remains in a failed state after performing the steps above, the card is likely faulty. 
Contact your Dell Support Provider to investigate the failure and arrange a repair or replacement.

To expedite your Service Request, confirm that 'Step 1 - 4' have been completed, then generate a new Support Bundle and attach it to the SR.