PowerEdge: Intel NIC and SONiC interface flapping issues
Summary: Customers experienced interface flapping issues with their Intel X710 NIC and PKR0R transceiver on SONiC 4.4.0 switches.
Symptoms
The interface flapping occurs during server reboots or OIR tests, causing operational disruptions. The problem persisted even after replacing the Intel X550 NIC with the Intel X710.
Firmware, Drivers, and OS Versions:
- Intel X710 NIC Firmware Version:
22.5.7 - Intel X710 NIC Driver Version:
2.5.11.0 - Intel E810 NIC Firmware Version:
23.0.8 - Intel E810 NIC Driver Version:
2.5.11.0 - Operating System Version:
SONiC 4.4.0
To identify the interface flapping issue in the switch logs, look for repeated messages indicating the port’s operational status changing frequently. For example:
NOTICE swss#orchagent: :- updatePortOperStatus: Port Eth1/1 oper state set from up to down
NOTICE swss#orchagent: :- updatePortOperStatus: Port Eth1/1 oper state set from down to up
Dell Part Numbers for NICs:
- Intel X710 NIC: Dell P/N
K5V44 - Intel E810 NIC: Dell P/N
VK88G - Broadcom BCM57416 NIC: Dell P/N
3TM39
Estimated Time for New Code: No new code is required as the issue was resolved through hardware replacement.
Cause
PKR0R Transceiver Requirements: The PKR0R transceivers require two W of power to operate effectively.
The Intel X710 NIC provides up to 1.5 W of power, which is insufficient for the PKR0R transceivers, leading to interface flapping issues.
Resolution
The issue was resolved by replacing the Intel X710 NIC with an Intel E810 NIC, which successfully eliminated the interface flapping.
Workarounds:
Shutting and then re-enabling the interface on the switch side after a server reboot or cable replacement.
Using alternative NICs such as Broadcom BCM57416, which also showed no flapping issues during lab tests.
OCP Cards: OCP cards are not affected by this flapping issue because they use a different architecture that is not susceptible to the same compatibility problems seen with the Intel X710 NIC and PKR0R transceiver.