SC Storage Notification: Cisco UCS VIC Firmware Causes Storage Center to Stop Responding

Issue:
Unexpected network credit depletion disrupts Fibre Channel (FC) storage traffic.

Symptom:
Operating System logs have many persistent SCSI abort commands for FCID 0xffffffff
Operating Systems using remote FC or FCoE storage may stop responding.

Prior to hang or aborts, there is an entry in VIC logs removing an FC MAC address such as: 

mcp.vnic_dev vnic17 vnic_dev_addr_del 0e:fc:00:06:22:0a

No ABORTS are seen at the VIC level.

At the OS, in vmkernel logs, there are repeated aborts. All aborts while in this condition shows a FCID of 0xffffffff

vmkernel: cpu33:397265)<7>fnic : 2 :: Abort Cmd called FCID 0xffffffff, LUN 0x1 TAG 5a flags 843

The host may stop responding and not recover without intervention. ESXi does NOT generate a purple screen.

Conditions:
Significant Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE) traffic

Affected Platforms:
All Storage Center Hardware Platforms may be affected by this issue where: 

• The VIC adapter firmware is version 4.2(3a)
  • 4.2(3a) VIC firmware is included with the 3.2(3a) UCS B-Series and C-Series software bundles
  • 4.2(3a) VIC firmware is included with the 3.1(3a) and 3.1(3b) UCS Standalone C-Series bundles

And

• Cisco UCS VIC hardware is one of the following:
  • Cisco UCS VIC 1340 modular LOM (UCSB-MLOM-40G-03)
  • Cisco UCS VIC 1380 mezzanine adapter (UCS-VIC-M83-8P)
  • Cisco UCS VIC 1385 Dual Port 40Gb QSFP+ CAN (UCSC-PCIE-C40Q-03)
  • Cisco UCS VIC 1387 Dual Port 40Gb QSFP CNA MLOM (UCSC-MLOM-C40Q-03)

Cisco UCS VIC firmware defect

Workaround:
Downgrade UCS VIC firmware.
For Cisco UCS B-Series Servers and Integrated C-Series Rack-Mount Servers, downgrade to a release prior to 3.2(3a) but at least 2.1(3) or later.
For Cisco UCS C-Series Standalone Rack-Mount Servers, downgrade to a release prior to 3.1(3a) but at least 2.1(3) or later.

Resolution:
This is resolved in Cisco UCS Manager Release 3.2(3d)B, 3.2(3d)C. See Cisco Software Advisory Notice.