We just got the new MX infrastructure, MX7000 chassis, MX740c servers, and MX9116n Fabric Switch Engines. While we will be switching to ethernet based storage later, we still have FC and thus, our plan is to use FCoE on the (esxi-)servers and use forward it as normal FC traffic to our Brocade FC switches on switch port 44 (configured for FC mode 16-4x)
Problem - I know little about FCoE configuration. I found this, but it didn't really help me too much: https://www.dell.com/support/home/us/en/04/product-support/product/networking-mx9116n/manuals
Do I need to venture into vFrabric configuration on the MX9116n? what FC feature mode, FPORT? NPG? Do I use FIP? Do I assign the vlan set for FCoE onto the CNA card, or use FIP alone?
Have anyone configured the following "path" and can share some config examples?
MX740c-with-CNA_FCoE <-> MX9116n <-> FC breakout cable 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage
I'm using the switches (two MX9116n) in full-switch mode btw.
Solved! Go to Solution.
Page 18 of the config guide may help with providing some of the steps, they are using smartfabrics though. https://downloads.dell.com/manuals/common/mx7000_fc_storage_networks_using_ethernet_ioms.pdf
Yes I found that. But I'm ot using smart fabric.. My uplinks at the moment is break out cables also for 4x10G ethernet (port 41) and 4x16G for FC (port 44). I will have a new look at it tomorrow anyway.
(This is @Olainen posting, found out i had two accounts. Will delete this account)
So there are no documentation (with examples) on setting up FCoE on MX9116n in full-switch mode?
What I've deducted so far is that the "fc domain-id" mode is not the mode I want, as I do want to do all zoning on my main Brocade switches (and I'm not sure if dell/mx <-> brocade is supported in "full fc switch mode" (aka F_PORT mode if I'm not mistaken?).
So it seems I need to use either "fip-snooping" or "fc mode npg"?
There is not any documentation yet, because it is a new product. FIP snooping should work for what you need.
With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers
So the FCoE/FC traffic will be like this:
MX740c w/CNA_FCoE (ports 1-16) <-> MX9116n <-> FC breakout 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage
For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.
My configuration for FCoE is so far like this:
interface vlan4092 description "FCoE-01-internal-vlan" no shutdown exit ! dcbx enable feature fc npg vfabric 1 name vFabric-1 vlan 4092 fcoe fcmap 0efc01 exit
interface vlan4093 description "FCoE-02-internal-vlan" no shutdown exit ! dcbx enable feature fc npg vfabric 2 name vFabric-2 vlan 4093 fcoe fcmap 0efc02 exit
FSE (both switches):
!# Break-out ethernet port-group 1/1/13 mode Eth 10g-4x ! !# Break-out fibrechannel port-group 1/1/16 mode FC 16g-4x class-map type network-qos class-Dot1p-3-FCoE match qos-group 3 class-map type queuing map-ETSQueue-0 match queue 0 class-map type queuing map-ETSQueue-3 match queue 3 exit ! trust dot1p-map map-Dot1pToGroups qos-group 0 dot1p 0-2,4-7 qos-group 3 dot1p 3 exit ! qos-map traffic-class map-GroupsToQueues queue 0 qos-group 0 queue 3 qos-group 3 exit ! policy-map policy-Input-TrustDot1p class class-trust trust dot1p exit exit ! policy-map type network-qos policy-Input-PFC class class-Dot1p-3-FCoE pause pfc-cos 3 exit exit ! policy-map type queuing policy-Output-BandwidthPercent class map-ETSQueue-0 bandwidth percent 68 class map-ETSQueue-3 bandwidth percent 32 exit exit ! system qos ets mode on trust-map dot1p map-Dot1pToGroups qos-map traffic-class map-GroupsToQueues exit
interface range ethernet 1/1/1-1/1/16 service-policy input type qos policy-Input-TrustDot1p service-policy input type network-qos policy-Input-PFC service-policy output type queuing policy-Output-BandwidthPercent priority-flow-control mode on vfabric 1 exit ! interface range fibrechannel 1/1/44:1-1/1/44:2 vfabric 1 exit exit
interface range ethernet 1/1/1-1/1/16 service-policy input type qos policy-Input-TrustDot1p service-policy input type network-qos policy-Input-PFC service-policy output type queuing policy-Output-BandwidthPercent priority-flow-control mode on vfabric 2 exit ! interface range fibrechannel 1/1/44:1-1/1/44:2 vfabric 2 exit
You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:
show fc switch show vfabric show npg devices brief show fcoe enode show fcoe fcf show fcoe sessions show fc statistics interface fibrechannel 1/1/44:1 show fc statistics interface fibrechannel 1/1/44:2 show fc statistics vfabric 1
Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):
OS10# show fcoe sessions
Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE MAC FC-ID PORT WWPN PORT WWNN
66:66:66:15:fe:7f Eth 1/1/15 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:02 05:8d:02 66:66:66:66:66:15:fe:7f 66:66:66:66:66:15:fe:7f
66:66:66:15:ff:e7 Eth 1/1/13 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:01 05:8d:01 66:66:66:66:66:15:ff:e7 66:66:66:66:66:15:ff:e7
66:66:66:15:ff:0f Eth 1/1/11 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:02 05:5c:02 66:66:66:66:66:15:ff:0f 66:66:66:66:66:15:ff:0f
66:66:66:0c:25:a7 Eth 1/1/1 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:01 05:5c:01 66:66:66:66:66:0c:25:a7 66:66:66:66:66:0c:25:a7
Hopefully this can help others get FCoE going on MX-infrastructure..
Great information, thanks for sharing.
Happy to help. I see all my three attempts to post got revived.. I'll see if I can clean it up.. The config is the same..
(Ugh. found out I had two accounts. This is @Olainen posting. Will delete this account, and keep my olainen account)
This explains a lot.. I couldn't change some of my posts.. I now see I have two account on here.. ugh.. Joffer77 and Olainen..
Is there a way to consolidate them?
I was able to remove the duplicate posts, but not combine your accounts.