XE9680 encounters an UEFI0031 PCIe downtrain issue during the boot process

Recently, after a sudden power cut, our XE9680 has been stopped ungracefully. And after the power restored, we started the server and found these message blocking the boot process:

I also checked the PCI link status in the system using

sudo lspci -vv | grep -E "^[0-9a-f]{2}:[0-9a-f]{2}\\.[0-9] |LnkCap:|LnkSta:"

and the result shows:

....

16:00.0 PCI bridge: Broadcom / LSI PEX890xx PCIe Gen 5 Switch (rev b0) (prog-if 00 [Normal decode])LnkCap:	Port #200, Speed 32GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 <32usLnkSta:	Speed 32GT/s, Width x8 (downgraded)

....

I attempted to unplug and replug the PCIe Raiser Card 1 and the two Mellanox InfiniteBand card, but the issue persists. Additionally, I followed the instructions on the UEFI0031 warning page, which involved removing the power cable, waiting for a few minutes, and then reconnecting it. I also updated the PCIe firmware, but the warning message still appears.

I searched and found a relevant discussion here. It mentions UEFI0031 and UEFI0067, but we’re experiencing a PCIe downtrain issue, not a PCIe link down failure.

Is there any solution to this problem or provide a way to bypass the block warning? I haven’t noticed any performance degradation yet, but it prevents the system from rebooting and booting. I need to manually press F1 in iDrac to proceed.

Firmware version:

• iDrac: 7.20.70.50
• BIOS: 2.7.5
• PCIe Switch: 4.218.1.0, A01