Dell Networking FCoE session issues due to LLDP packets sent by VMs.

• Lost host communication to FCoE environment
• Storage communication stops responding or flaps all the time.
• In a stable environment a COMPLETE OUTAGE of all FCoE vlans may occur when a manual action is taken that causes LLDP to renumerate; for example, adding a VLAN or disabling LLDP on the vDS. 

This article is intended to provide an additional protection for FCoE environments where a VM sends LLDP packets to the (FIP/FSB) switch by accident and that breaks the FCoE connection to the switch. During the problem state, you see this message below in the switch’s syslog.

Error message: "LLDP_MULTIPLE_PEER_DETECTED: DCBX operationally disabled due to more than one PEER being present on interface" when enabling LLDP from the OS)

Warning Message:
-------------------------------------
LLDP_MULTIPLE_PEER_DETECTED: DCBX
-------------------------------------
Two possible issues you can see:

1. VMware vDS is sending and receiving or only sending LLDP packets.
   1. Usually, it happens when the LLDP configuration on VMware vDS is set as "Both(listen and advertise)" or is set as "advertise.”
2. VMs are sending LLDP packets through VMware vDS.
   1. If the VM’s operation system is sending LLDP packets to advertise its presence that breaks the FCoE connectivity due to the nature of FCoE operation.

Another indication that the switch is seeing multiple LLDP neighbors and is in the problem state:
 

The network may present itself as perfectly stable even when "show lldp neighbors" indicates that the switch is in the problem state. However, if a manual action like adding a VLAN or disabling LLDP on the vDS is taken, it may trigger a complete outage on all FCoE sessions.

Why Would This Happen?
FCoE relies on DCBx, and DCBx is a protocol that runs on LLDP. 

"DCBX uses Link Layer Discovery Protocol (LLDP) to exchange parameters between two link peers. LLDP is a unidirectional protocol. It advertises connectivity and management information about the local station to adjacent stations on the same IEEE 802 LAN." - https://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-protocol-1108-v1.01.pdf
"DCBX is expected to operate over a point-to-point link. If multiple LLDP neighbors are detected, then DCBX behaves as if the peer’s DCBX TLVs are not present until the multiple LLDP neighbor condition is no longer present. An LLDP neighbor is identified by its logical MAC Service Access Identifier (MSAP). The logical MSAP is a concatenation of the chassis ID and port ID values transmitted in the LLDPDU." - https://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-protocol-1108-v1.01.pdf
Any change that causes LLDP to renumerate may cause FCoE's TLV functionality to immediately break.

Option 1) change LLDP mode on the vDS per Broadcom KB.
Option 2) What we are going to do is to add a VMware vDS filtering policy to block LLDP packet from VMs, this configuration alone blocks any LLDP packet from any VM under that vDS Port-Group. *This issue can be seen with VMware standard vSwitch as well, but no work round or filtering option is available. The customer must make sure their VMs are LLDP free.
 

 Procedure

 Reference: VMware traffic filtering guide

1. Locate a distributed port group or an uplink port group in the vSphere Client.

1. Select a distributed switch and click the Networks tab.
2. Click Distributed Port Groups to see the list of distributed port groups or click Uplink Port Groups to see the list of uplink port groups.

2. Click a distributed port group or an uplink port group and select the Configure tab.
3. Under Settings, select Traffic Filtering And Marking.
4. Click the Enable and reorder button.

5. Click Enable all traffic rules.
6. Click OK.

7. Click ADD button.

VMware MAC Traffic Qualifier reference guide.

8. In the Rule window, set the parameters below.
   1. "Name" the rule to describe the action.
   2. On "Action" field, select "Drop."
   3. On "Traffic direction," select "Ingress."
   4. Click "MAC" tab.
      1. Enable the "Enable qualifier" checkbox.
      2. On "EtherType" field:
         1. Select "IS."
         2. Select "Custom."
         3. Write "88CC" as the LLDP ethertype.
   5. Click OK.

9. At the end, you should have the configuration below.

10. Check if any LLDP packet is being blocked from the selected VM.
    1. Click Ports tab.
       1. Select vDS Port ID where is the VM.
       2. Click Statistics tab.
          1. Check "Dropped - Ingress Packets"

   

11 That concludes the configuration.