PowerFlex Single SDC Having Performance Issues With Mellanox 25gbit NICs

Scenario
Each node was running fio with 128 K 50r/50w, eight threads, and iodepth of 16 (256 threads). The problematic SDC would max out at 4GBs where the other four nodes would hit ~6GBs each. When looking at the performance monitor of vCenter it shows that vmnic1 is less utilized than vmnic3, where-as the other hosts w/ the same fio job show that the utilization is almost 100% equal.

Symptoms
The utilization of multiple NICs on a single vSwitch within ESXi show unbalanced. When reviewing the "nicinfo.sh.txt" file from the vmsupport pulled from the ESXi host that is having the problems we can see that there are physical errors showing, with vmnic1 being the largest culprit. 

The example output is from ESXi 6.7, and older versions may not contain all the same data or it may look differently. 

$ grep -i -e err -e vmnic -e discard -e crc -e drop -e retr -e wake nicinfo.sh.txt | grep -v ": 0"

NIC:  vmnic0

   Name: vmnic0

   Supports Wakeon: true

   Wakeon: MagicPacket(tm)

NIC statistics for vmnic0:

   Total receive errors: 9

   Receive length errors: 9

txSwQueueWake: 7

rxDiscardsPhy: 289

TX1_wake: 2

TX2_wake: 4

TX3_wake: 1

NIC:  vmnic1

   Name: vmnic1

   Supports Wakeon: true

   Wakeon: MagicPacket(tm)

NIC statistics for vmnic1:

txSwQueueWake: 3027034

rxDiscardsPhy: 16773308

TX0_wake: 3027034

NIC:  vmnic2

   Name: vmnic2

   Supports Wakeon: false

   Wakeon: None

NIC statistics for vmnic2:

   Total receive errors: 16

   Receive length errors: 16

NIC:  vmnic3

   Name: vmnic3

   Supports Wakeon: false

   Wakeon: None

NIC statistics for vmnic3:

txSwQueueWake: 153

rxDiscardsPhy: 804

TX0_wake: 150

TX2_wake: 1

TX3_wake: 2

Impact

Reduced throughput and overall performance from a single SDC

This problem had something to do with the physical layer, this could be a failed SFP, NIC port, and possibly the cable itself. In this case it the connection was Twinax, it turned out there was a bad cable causing the packet drops and reduced performance. There was also a possibility this could have been an oversubscribed port possibly on the switch side as well. 

Replace the faulty physical component.