PowerEdge: Mellanox CX-5: unrecoverable hardware error on POST

Summary: Dell customers may report an unrecoverable hardware error on POST with configurations including a certain mix of Mellanox firmware, BIOS, and iDRAC versions. While the issue was reported on VMware with CX-5 cards, other Dell/Mellanox hardware may be affected. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Once the issue is encountered, the device is in a non-functional state and can lead to an ESXi host crash (PSOD).
Issue signatures may include (but not limited to) the below boot console and vmkernel log messages:

Boot system with OS + CX5. Console reports the following:

2021-12-21T10:29:46.037Z cpu22:2098137)WARNING:  device's health compromised
2021-12-21T10:29:46.037Z cpu22:2098137) assertVar[0] 0x00000002
2021-12-21T10:29:46.037Z cpu22:2098137) assertVar[1] 0x000ddf7c
2021-12-21T10:29:46.037Z cpu22:2098137) assertVar[2] 0x00000000
2021-12-21T10:29:46.037Z cpu22:2098137) assertVar[3] 0x00000000
2021-12-21T10:29:46.037Z cpu22:2098137) assertVar[4] 0x00000000
2021-12-21T10:29:46.037Z cpu22:2098137) assertExitPtr 0x00804b98
2021-12-21T10:29:46.037Z cpu22:2098137) assertCallra 0x00804e40
2021-12-21T10:29:46.037Z cpu22:2098137) firmwareVersion 0x101a1770
2021-12-21T10:29:46.037Z cpu22:2098137) hwId 0x0000020d
2021-12-21T10:29:46.037Z cpu22:2098137) iriscIndex 5
2021-12-21T10:29:46.037Z cpu22:2098137) synd 0x8: unrecoverable hardware error
2021-12-21T10:29:46.037Z cpu22:2098137) extSynd 0x0087
2021-12-21T10:29:46.037Z cpu38:2098149)WARNING:  handling bad device here


Also, in the OS logs: 

# grep "unrecoverable hardware error" /var/log/messages
2021-12-17T05:21:35.159473-06:00 <0.7> MN-R640-77-1(id1) /boot/kernel.amd64/kernel: mlx5_core: INFO: synd 0x8: unrecoverable hardware error


 

 

 

Cause

The BMC opens a channel over SMBUS and at some point moves the media to PCIe over VDM for sideband communication, and the ConnectX-5 device does not handle the transition.

Resolution

Dell engineering is aware of this issue, Do not replace hardware.

This behavior is resolved in Mellanox firmware 16.32.20.04. Have the customer update to this firmware revision or newer.

Affected Products

Mellanox Family of Adapters, VMware ESXi 7.x

Products

PowerEdge XR2, Poweredge C4140, PowerEdge C6420, PowerEdge C6520, PowerEdge C6525, PowerEdge MX5016s, PowerEdge MX740C, PowerEdge MX750c, PowerEdge MX840C, PowerEdge R350, PowerEdge R440, PowerEdge R450, PowerEdge R540, PowerEdge R550, PowerEdge R640 , PowerEdge R6415, PowerEdge R650, PowerEdge R650xs, PowerEdge R6515, PowerEdge R6525, PowerEdge R740, PowerEdge R740XD, PowerEdge R740XD2, PowerEdge R7415, PowerEdge R7425, PowerEdge R750, PowerEdge R750XA, PowerEdge R750xs, PowerEdge R7515, PowerEdge R7525, PowerEdge R840, PowerEdge R940, PowerEdge R940xa, PowerEdge T350, PowerEdge T440, PowerEdge T550, PowerEdge T640, PowerEdge XE2420, PowerEdge XE7420, PowerEdge XE7440, PowerEdge XE8545, PowerEdge XR11, PowerEdge XR12, PowerFlex appliance R650, PowerFlex appliance R6525, Powerflex appliance R750, VMware ESXi 8.x, PowerFlex appliance R640, PowerFlex appliance R740XD, PowerFlex appliance R7525, PowerFlex appliance R840 ...
Article Properties
Article Number: 000195461
Article Type: Solution
Last Modified: 13 مايو 2026
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.