Olainen
1 Nickel

MX9116n FCoE-FC configuration?

Jump to solution

We just got the new MX infrastructure, MX7000 chassis, MX740c servers, and MX9116n Fabric Switch Engines. While we will be switching to ethernet based storage later, we still have FC and thus, our plan is to use FCoE on the (esxi-)servers and use forward it as normal FC traffic to our Brocade FC switches on switch port 44 (configured for FC mode 16-4x)

Problem - I know little about FCoE configuration. I found this, but it didn't really help me too much: https://www.dell.com/support/home/us/en/04/product-support/product/networking-mx9116n/manuals

Do I need to venture into vFrabric configuration on the MX9116n? what FC feature mode, FPORT? NPG? Do I use FIP? Do I assign the vlan set for FCoE onto the CNA card, or use FIP alone?

Have anyone configured the following "path" and can share some config examples?

MX740c-with-CNA_FCoE <-> MX9116n <-> FC breakout cable 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage

I'm using the switches (two MX9116n) in full-switch mode btw.

Labels (3)
1 Solution

Accepted Solutions
Highlighted
Olainen
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers 

Environment:

  • MX7000 chassis with 2 x MX9116n Fabric Switch Engines ("FSE")
  • MX740c Servers with QL41262 CNA cards (4 servers so far in my chasses, all 8 will get CNA soon)
  • Brocade FC fabrics wiht HUS-VM SAN behind

So the FCoE/FC traffic will be like this:

MX740c w/CNA_FCoE (ports 1-16) <-> MX9116n <-> FC breakout 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage

For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.

My configuration for FCoE is so far like this:

FSE-A1:

interface vlan4092
 description "FCoE-01-internal-vlan"
 no shutdown
 exit
!
dcbx enable
feature fc npg
vfabric 1
 name vFabric-1
 vlan 4092
 fcoe fcmap 0efc01
 exit

FSE-A2:

interface vlan4093
 description "FCoE-02-internal-vlan"
 no shutdown
 exit
!
dcbx enable
feature fc npg
vfabric 2
 name vFabric-2
 vlan 4093
 fcoe fcmap 0efc02
 exit

FSE (both switches):

!# Break-out ethernet
port-group 1/1/13
 mode Eth 10g-4x
!
!# Break-out fibrechannel
port-group 1/1/16
 mode FC 16g-4x

class-map type network-qos class-Dot1p-3-FCoE
 match qos-group 3
class-map type queuing map-ETSQueue-0
 match queue 0
class-map type queuing map-ETSQueue-3
 match queue 3
exit
!
trust dot1p-map map-Dot1pToGroups
 qos-group 0 dot1p 0-2,4-7
 qos-group 3 dot1p 3
 exit
!
qos-map traffic-class map-GroupsToQueues
 queue 0 qos-group 0
 queue 3 qos-group 3
 exit
!
policy-map policy-Input-TrustDot1p
 class class-trust
 trust dot1p
 exit
 exit
!
policy-map type network-qos policy-Input-PFC
 class class-Dot1p-3-FCoE
 pause
 pfc-cos 3
 exit
 exit
!
policy-map type queuing policy-Output-BandwidthPercent
 class map-ETSQueue-0
 bandwidth percent 68
 class map-ETSQueue-3
 bandwidth percent 32
 exit
 exit
!
system qos
 ets mode on
 trust-map dot1p map-Dot1pToGroups 
 qos-map traffic-class map-GroupsToQueues
 exit

FSE-A1:

interface range ethernet 1/1/1-1/1/16
service-policy input type qos policy-Input-TrustDot1p
service-policy input type network-qos policy-Input-PFC
service-policy output type queuing policy-Output-BandwidthPercent
priority-flow-control mode on
vfabric 1
exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
vfabric 1
exit
exit

FSE-A2:

interface range ethernet 1/1/1-1/1/16
 service-policy input type qos policy-Input-TrustDot1p
 service-policy input type network-qos policy-Input-PFC
 service-policy output type queuing policy-Output-BandwidthPercent
 priority-flow-control mode on
 vfabric 2
 exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
 vfabric 2
 exit

You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:

show fc switch
show vfabric
show npg devices brief
show fcoe enode
show fcoe fcf
show fcoe sessions  
show fc statistics interface fibrechannel 1/1/44:1
show fc statistics interface fibrechannel 1/1/44:2
show fc statistics vfabric 1

Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):

OS10# show fcoe sessions
Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE MAC FC-ID PORT WWPN PORT WWNN
---------------------------------------------------------------------------------------------------------------------------------------------------------
66:66:66:15:fe:7f Eth 1/1/15 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:02 05:8d:02 66:66:66:66:66:15:fe:7f 66:66:66:66:66:15:fe:7f
66:66:66:15:ff:e7 Eth 1/1/13 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:01 05:8d:01 66:66:66:66:66:15:ff:e7 66:66:66:66:66:15:ff:e7
66:66:66:15:ff:0f Eth 1/1/11 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:02 05:5c:02 66:66:66:66:66:15:ff:0f 66:66:66:66:66:15:ff:0f
66:66:66:0c:25:a7 Eth 1/1/1 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:01 05:5c:01 66:66:66:66:66:0c:25:a7 66:66:66:66:66:0c:25:a7

Hopefully this can help others get FCoE going on MX-infrastructure..

13 Replies
Moderator
Moderator

Re: MX9116n FCoE-FC configuration?

Jump to solution

Hi,

Page 18 of the config guide may help with providing some of the steps, they are using smartfabrics though. https://downloads.dell.com/manuals/common/mx7000_fc_storage_networks_using_ethernet_ioms.pdf

Thanks,
Josh Craig
Dell EMC Enterprise Support Services
Get support on Twitter @DellCaresPRO
0 Kudos
Joffer77
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

Yes I found that. But I'm ot using smart fabric.. My uplinks at the moment is break out cables also for 4x10G ethernet (port 41) and 4x16G for FC (port 44). I will have a new look at it tomorrow anyway.

(This is @Olainen posting, found out i had two accounts. Will delete this account)

0 Kudos
Olainen
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

So there are no documentation (with examples) on setting up FCoE on MX9116n in full-switch mode?

What I've deducted so far is that the "fc domain-id" mode is not the mode I want, as I do want to do all zoning on my main Brocade switches (and I'm not sure if dell/mx <-> brocade is supported in "full fc switch mode" (aka F_PORT mode if I'm not mistaken?).

So it seems I need to use either "fip-snooping" or "fc mode npg"? 

0 Kudos
Moderator
Moderator

Re: MX9116n FCoE-FC configuration?

Jump to solution

There is not any documentation yet, because it is a new product. FIP snooping should work for what you need.

Thanks,
Josh Craig
Dell EMC Enterprise Support Services
Get support on Twitter @DellCaresPRO
0 Kudos
Highlighted
Olainen
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers 

Environment:

  • MX7000 chassis with 2 x MX9116n Fabric Switch Engines ("FSE")
  • MX740c Servers with QL41262 CNA cards (4 servers so far in my chasses, all 8 will get CNA soon)
  • Brocade FC fabrics wiht HUS-VM SAN behind

So the FCoE/FC traffic will be like this:

MX740c w/CNA_FCoE (ports 1-16) <-> MX9116n <-> FC breakout 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage

For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.

My configuration for FCoE is so far like this:

FSE-A1:

interface vlan4092
 description "FCoE-01-internal-vlan"
 no shutdown
 exit
!
dcbx enable
feature fc npg
vfabric 1
 name vFabric-1
 vlan 4092
 fcoe fcmap 0efc01
 exit

FSE-A2:

interface vlan4093
 description "FCoE-02-internal-vlan"
 no shutdown
 exit
!
dcbx enable
feature fc npg
vfabric 2
 name vFabric-2
 vlan 4093
 fcoe fcmap 0efc02
 exit

FSE (both switches):

!# Break-out ethernet
port-group 1/1/13
 mode Eth 10g-4x
!
!# Break-out fibrechannel
port-group 1/1/16
 mode FC 16g-4x

class-map type network-qos class-Dot1p-3-FCoE
 match qos-group 3
class-map type queuing map-ETSQueue-0
 match queue 0
class-map type queuing map-ETSQueue-3
 match queue 3
exit
!
trust dot1p-map map-Dot1pToGroups
 qos-group 0 dot1p 0-2,4-7
 qos-group 3 dot1p 3
 exit
!
qos-map traffic-class map-GroupsToQueues
 queue 0 qos-group 0
 queue 3 qos-group 3
 exit
!
policy-map policy-Input-TrustDot1p
 class class-trust
 trust dot1p
 exit
 exit
!
policy-map type network-qos policy-Input-PFC
 class class-Dot1p-3-FCoE
 pause
 pfc-cos 3
 exit
 exit
!
policy-map type queuing policy-Output-BandwidthPercent
 class map-ETSQueue-0
 bandwidth percent 68
 class map-ETSQueue-3
 bandwidth percent 32
 exit
 exit
!
system qos
 ets mode on
 trust-map dot1p map-Dot1pToGroups 
 qos-map traffic-class map-GroupsToQueues
 exit

FSE-A1:

interface range ethernet 1/1/1-1/1/16
service-policy input type qos policy-Input-TrustDot1p
service-policy input type network-qos policy-Input-PFC
service-policy output type queuing policy-Output-BandwidthPercent
priority-flow-control mode on
vfabric 1
exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
vfabric 1
exit
exit

FSE-A2:

interface range ethernet 1/1/1-1/1/16
 service-policy input type qos policy-Input-TrustDot1p
 service-policy input type network-qos policy-Input-PFC
 service-policy output type queuing policy-Output-BandwidthPercent
 priority-flow-control mode on
 vfabric 2
 exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
 vfabric 2
 exit

You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:

show fc switch
show vfabric
show npg devices brief
show fcoe enode
show fcoe fcf
show fcoe sessions  
show fc statistics interface fibrechannel 1/1/44:1
show fc statistics interface fibrechannel 1/1/44:2
show fc statistics vfabric 1

Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):

OS10# show fcoe sessions
Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE MAC FC-ID PORT WWPN PORT WWNN
---------------------------------------------------------------------------------------------------------------------------------------------------------
66:66:66:15:fe:7f Eth 1/1/15 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:02 05:8d:02 66:66:66:66:66:15:fe:7f 66:66:66:66:66:15:fe:7f
66:66:66:15:ff:e7 Eth 1/1/13 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:01 05:8d:01 66:66:66:66:66:15:ff:e7 66:66:66:66:66:15:ff:e7
66:66:66:15:ff:0f Eth 1/1/11 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:02 05:5c:02 66:66:66:66:66:15:ff:0f 66:66:66:66:66:15:ff:0f
66:66:66:0c:25:a7 Eth 1/1/1 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:01 05:5c:01 66:66:66:66:66:0c:25:a7 66:66:66:66:66:0c:25:a7

Hopefully this can help others get FCoE going on MX-infrastructure..

Moderator
Moderator

Re: MX9116n FCoE-FC configuration?

Jump to solution

Great information, thanks for sharing.

Thanks,
Josh Craig
Dell EMC Enterprise Support Services
Get support on Twitter @DellCaresPRO
0 Kudos
Joffer77
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

Happy to help. I see all my three attempts to post got revived.. I'll see if I can clean it up.. The config is the same.. 

(Ugh. found out I had two accounts. This is @Olainen posting. Will delete this account, and keep my olainen account)

0 Kudos
Olainen
1 Nickel

Re: MX9116n FCoE-FC configuration?

Jump to solution

This explains a lot.. I couldn't change some of my posts.. I now see I have two account on here.. ugh.. Joffer77 and Olainen.. 

Is there a way to consolidate them? 

0 Kudos
Moderator
Moderator

Re: MX9116n FCoE-FC configuration?

Jump to solution

I was able to remove the duplicate posts, but not combine your accounts.

Thanks,
Josh Craig
Dell EMC Enterprise Support Services
Get support on Twitter @DellCaresPRO
0 Kudos