PowerEdge: Intel NIC and SONiC interface flapping issues

The interface flapping occurs during server reboots or OIR tests, causing operational disruptions. The problem persisted even after replacing the Intel X550 NIC with the Intel X710.

Firmware, Drivers, and OS Versions:

• Intel X710 NIC Firmware Version: 22.5.7
• Intel X710 NIC Driver Version: 2.5.11.0
• Intel E810 NIC Firmware Version: 23.0.8
• Intel E810 NIC Driver Version: 2.5.11.0
• Operating System Version: SONiC 4.4.0

To identify the interface flapping issue in the switch logs, look for repeated messages indicating the port’s operational status changing frequently. For example:

NOTICE swss#orchagent: :- updatePortOperStatus: Port Eth1/1 oper state set from up to down

NOTICE swss#orchagent: :- updatePortOperStatus: Port Eth1/1 oper state set from down to up

Dell Part Numbers for NICs:

• Intel X710 NIC: Dell P/N K5V44
• Intel E810 NIC: Dell P/N VK88G
• Broadcom BCM57416 NIC: Dell P/N 3TM39

Estimated Time for New Code: No new code is required as the issue was resolved through hardware replacement.

PKR0R Transceiver Requirements: The PKR0R transceivers require two W of power to operate effectively.

The Intel X710 NIC provides up to 1.5 W of power, which is insufficient for the PKR0R transceivers, leading to interface flapping issues.

The issue was resolved by replacing the Intel X710 NIC with an Intel E810 NIC, which successfully eliminated the interface flapping.

Workarounds:
Shutting and then re-enabling the interface on the switch side after a server reboot or cable replacement.
Using alternative NICs such as Broadcom BCM57416, which also showed no flapping issues during lab tests.

OCP Cards: OCP cards are not affected by this flapping issue because they use a different architecture that is not susceptible to the same compatibility problems seen with the Intel X710 NIC and PKR0R transceiver.