37 Posts
2
21213
MX9116n FCoE-FC configuration?
We just got the new MX infrastructure, MX7000 chassis, MX740c servers, and MX9116n Fabric Switch Engines. While we will be switching to ethernet based storage later, we still have FC and thus, our plan is to use FCoE on the (esxi-)servers and use forward it as normal FC traffic to our Brocade FC switches on switch port 44 (configured for FC mode 16-4x)
Problem - I know little about FCoE configuration. I found this, but it didn't really help me too much: https://www.dell.com/support/home/us/en/04/product-support/product/networking-mx9116n/manuals
Do I need to venture into vFrabric configuration on the MX9116n? what FC feature mode, FPORT? NPG? Do I use FIP? Do I assign the vlan set for FCoE onto the CNA card, or use FIP alone?
Have anyone configured the following "path" and can share some config examples?
MX740c-with-CNA_FCoE <-> MX9116n <-> FC breakout cable 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage
I'm using the switches (two MX9116n) in full-switch mode btw.
Olainen
37 Posts
2
January 28th, 2019 05:00
With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers :BigSmile:
Environment:
So the FCoE/FC traffic will be like this:
For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.
My configuration for FCoE is so far like this:
FSE-A1:
FSE-A2:
FSE (both switches):
FSE-A1:
FSE-A2:
You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:
Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):
Hopefully this can help others get FCoE going on MX-infrastructure..
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
January 21st, 2019 10:00
Hi,
Page 18 of the config guide may help with providing some of the steps, they are using smartfabrics though. https://downloads.dell.com/manuals/common/mx7000_fc_storage_networks_using_ethernet_ioms.pdf
Joffer77
7 Posts
0
January 21st, 2019 10:00
Yes I found that. But I'm ot using smart fabric.. My uplinks at the moment is break out cables also for 4x10G ethernet (port 41) and 4x16G for FC (port 44). I will have a new look at it tomorrow anyway.
(This is @Olainen posting, found out i had two accounts. Will delete this account)
Olainen
37 Posts
0
January 23rd, 2019 02:00
So there are no documentation (with examples) on setting up FCoE on MX9116n in full-switch mode?
What I've deducted so far is that the "fc domain-id" mode is not the mode I want, as I do want to do all zoning on my main Brocade switches (and I'm not sure if dell/mx <-> brocade is supported in "full fc switch mode" (aka F_PORT mode if I'm not mistaken?).
So it seems I need to use either "fip-snooping" or "fc mode npg"?
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
January 23rd, 2019 09:00
There is not any documentation yet, because it is a new product. FIP snooping should work for what you need.
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
January 29th, 2019 09:00
Great information, thanks for sharing.
Joffer77
7 Posts
0
January 29th, 2019 13:00
Happy to help. I see all my three attempts to post got revived.. I'll see if I can clean it up.. The config is the same..
(Ugh. found out I had two accounts. This is @Olainen posting. Will delete this account, and keep my olainen account)
Olainen
37 Posts
0
January 30th, 2019 01:00
This explains a lot.. I couldn't change some of my posts.. I now see I have two account on here.. ugh.. Joffer77 and Olainen..
Is there a way to consolidate them?
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
January 30th, 2019 08:00
I was able to remove the duplicate posts, but not combine your accounts.
Olainen
37 Posts
0
February 5th, 2019 01:00
Ok. I'll just stop using the other account :)
VAL24
8 Posts
0
May 16th, 2019 21:00
Hi Olainen
Did you have to assign vmacs and wwns to the identity pool before this? Is that an absolute pre-requisite?
I have put together the config but not getting fcoe sessions. I have not created any idenity pools yet.
KVS
VAL24
8 Posts
0
May 17th, 2019 02:00
Hi Olainen
Even after adding the identity pools am not seeing the sessions. What are the troubleshooting steps?
Val
VAL24
8 Posts
0
May 19th, 2019 15:00
So I managed to get one site working with FCoE. looks like you have to reboot the server to see the fcoe sessions. below is a sample config of the interface.
interface ethernet1/1/9
description "LINK to Hosts"
no shutdown
fec off
switchport mode trunk
switchport access vlan 1
service-policy input type network-qos policy-Input-PFC
service-policy input type qos policy-Input-TrustDot1p
service-policy output type queuing policy-Output-BandwidthPercent
vfabric 2
priority-flow-control mode on
!
I partitioned the QL41262 with 2x FCoE and 2x LAN per hba. Theres new firmware on the Force10 site which I will get and update the Mx9116n tomorrow. theres also firmware on the dellemc support site but apparently that can break the switch.
I have seen on the S4148 config that ets mode on. not 100% sure if that improves anything so didnt include it.
This switch is so new theres literally no help online for it. Dell should have rolled out the training way ahead. The chassis management is also another story with the new update.
Olainen
37 Posts
0
June 25th, 2019 01:00
I'm running my MX9116n's on v10.4.0E.R4S and some is on v10.4.0E.R4S.P1. Firmware for QL41262 should be updated. Which OS are you running? ESXi? Windows? Linux? I'm on ESXi 6.7 and had to update the driver as well. I do think the latest ISOs from Dell contains the latest recommended driver (3.9.31.2) (there is a newer one too but was not recommended by Dell).
Olainen
37 Posts
0
June 25th, 2019 02:00
Yes I've got ETS on, both in system qos and on the server ports. Seems you copied the qos/policy-maps/class-maps so that should be fine. I see my config on the first page is a bit different than my current config. I would add these settings to my original config (i wasn't able to edit the post):
NB! Only showing for the first FSE, so change vfabric on the second
Global settings:
One server facing port: