ISL a new Cisco Fiber switch to existing Fabric

Yes you'll need to have VSAN70 in place on the new switch for it merge with the current fabric's VSAN 70. It sounds like you take the defaults on lot of the settings so I don't think you will have issues with domain ids, allowed lists or anything of that nature. I believe trunks default to allowing all VSANs to travel across them so you should be good there.

Thanks Hersh....One last quick question... When you create your VSAN (I use device manager)..do you pick the port in the "Interface Members" field or leave it blank?  If you pick the ports, then which port would you pick?  The one you use for ISL only or all the ports..  Sorry for my ignorance, but this is not something I do on a everyday basis....thanks...

Inferface Members is the bottom field.

Sorry for the delay. Is the ISL already up and running? If so I would avoid changing that port's VSAN since it will disrupt the ISL (probably wouldn't cause any harm) plus you mentioned you like to keep ISLs in VSAN 1, thats what we do here as well. You can select all the ports you want to use VSAN 70 in that Section if you want. I'm unsure of what it would do if that port was already in use in another VSAN, so it might be safer to assign the VSANs afterwards on a port by port basis.

I usually do it on a port by port basis since we have a couple different VSANs on some of our switches.

No Hersh...I have not implemented this yet.. being asked to put this project on hold for now...I think I know what its purpose is for now...If you create a VSAN (say 70 in this case) you can leave this field blank, choose a certain number of port, or select all of them (except the ISL ports).  It's only used for default VSAN when you configure each of those ports later.  For whatever ports you choose in this field when you create this VSAN, when you configure those ports they'll have that VSAN as default.

No it will not allow you to include the ports you already used for ISL (you'll get an error).  Basically choosing the ports for that field will eliminate 1 step (choosing which VSAN it belongs to) when you configure those ports.  Now that I know the rest of the ports will be used for VSAN70, I can choose all of them (except the ISL ones) and I should be good.  And this can be changed later too...Hope this info be useful for you later.  Thanks for responding...

Hi I have a question around the fabric reconfiguration(s).

When you introduce a new switch into an existing fabric that's 1 x fabric reconfiguration (which is disruptive) regardless of the switch priority right? Or is the fabric reconfiguration dependant on the switch priority of the switch you are introducing? Example if the new switch priority is that so it becomes principal switch.

Second, when you demote and remove (decommission) an existing principal switch that is certainly a fabric reconfiguration and restart.

Lastly I'm looking for some documentation that articulates this but not having a lot of luck.

Cheers!

the new switch should be of a lower priority than the existing switch, else it will become the master. So when you add a fresh switch, make sure to demote its priority so that it becomes the subordinate switch to avoid any reconfiguration.

Consider to verify that the configured domain IDs and the runtime domain IDs are matching.

The default would be to have configured domain ID = preferred / zero. If your configured Domain IDs are preferred / zero, you should set the configured Domain ID to match the runtime Domain ID and to be static non-disruptively.

Doing a fabric merge without changing Domain IDs will cause a non disruptive build fabric (BF), changing a domain ID will cause a disruptive reconfigure fabric (RCF).

The principal switch should be in the core. If you are using the defaults, all switches will have a configured priority of 128. The current principal will have a running priority of 2. If you merge a new switch into the fabric, the principal switch will remain its role. If there is a need to change the principal, you need to set the prio to 1 non-disruptively. After the build fabric change the prio of the principal back to a higher prio than the default. For example: principal prio = 10, secondary principal prio = 20.

The switch firmware should be upgraded to EMC target before doing the merge.

Consider creating a portChannel having mode = active.

We are planning to ISL MDS 9710, which has an active zoneset on vsan 3 to an MDS9222i which has an active zoneset on vsan 2. Both the zonesets have different members. Our aim is to

1) ISL both the switches. While doing this I want to have both vsan 2 and vsan 3 on both the switches, with its respective zoneset

2) Once I have achieved that, I would be migrating all the components of vsan 2 to vsan 3. as our aim is to start sharing SAN storage between these servers.

To achieve this second step, I thought I will copy the zoning and fcalias configuration from the active zoneset in MDS9222i and add it to the active zoneset in MDS9710.

for the ISL

The steps that I have now in mind is

1. Create vsan 3 on mds9710

2. Create vsan 2 on mds9222i

3. change configured priority to 10 on mds 9710 for vsans 1, 2 and 3 and do an fcdomain restart

4. Change zone to enhanced mode in both switches

5. Set zoneset distribution to full on both switches (we use fcalias and not device alias)

6. Set persistent domain id and fcid on both the switches

6. Do an fcdomain restart on all vsans for mds9710 to change the configured priority to running priority.

7. Create port-channel in active mode. Here since we are using 9710 and 9222i is there any speed I need to set it to

or is it enough to keep it as switchport mode E, rate-mode dedicated, trunk mode on, ? and then I will change it to active mode.

8. do a show cfs peers to check if it has merged.

Please find my suggestion - I reordered the tasks and added some notes - bellow:

A. Optimize the existing configuration

6. Set persistent domain id and fcid on both the switches

Consider using the active runtime domain IDs – If the domain id will not change, the FCIDs will not change.

5. Set zoneset distribution to full on both switches

4. Change zone to enhanced mode in both switches

3. Change configured priority to 10 on mds 9710 for vsans 1 and 3…

6. Do an fcdomain restart on all vsans for mds9710 to change the configured priority to running priority.

If there is no domain ID change, you will be able to do a non-disruptive fcdomain restart / built fabric (bf).

B. Configure the Port Channel

7. Create port-channel in active mode…

…keep it as switchport mode E, rate-mode dedicated, trunk mode on…

I would prefer to set a fixed speed and restrict the trunk allowed list to allow vsan 1, vsan 3

8. do a show cfs peers to check if vsan 1 has merged.

C. Create the VSAN and Migrate

1. Create vsan 3 on mds9222i – use the running domain id of vsan 2.

8. do a show cfs peers to check if vsan 3 has merged.

2) …copy the zoning and fcalias configuration from the active zoneset in MDS9222i / vsan 2 and add it to the active zoneset in MDS9710 / vsan 3…

Copy the FCID configuration from vsan 2 to vsan 3 too would provide the least possible impact for attached devices

2) …migrating all the components of vsan 2 to vsan 3…

You can use the device manager to move all ports from vsan 2 to vsan 3 at once - the ports will stay online!

If the FCIDs are copied too, the devices will have the least possible impact.

I hope this will help?

why are you trunking VSAN 1 ?

Thank you so much for the reply.

I have a question about creating VSANs. I have vsan 3 on mds9710 and vsan 2 on mds9222i. So I believe we will need to create an empty vsan 2 on mds9710 and empty vsan 3 on mds9222i.

this is for initial merge, and then I will combine vsan 2 and vsan 3.

Next question is about speed. Which speed should I chose?

Because the mds9222i has most of it at automax 4g and mds9710 is it automax 16g. also I am using a 4 port port-channel for the ISL

dynamo, we are not trunking vsan 1 . but I thought we should do everything that we do on active vsans on vsan 1 too?

Also the ISL ports remain on vsan 1? am I right?

yeah, you are not using VSAN 1, no need to trunk it.  I find it to be best practice to put "ISL" ports into VSAN 1, because VSAN 1 cannot be deleted (even if you wanted to) so it prevents "oops" kind of errors. So ISL ports will be in VSAN 1 and then you will trunk VSAN 2 and 3.

over the years i've talked to support techs at TAC and other engineers at Cisco, they have always told me not to trunk VSAN 1. I know it looks ugly and segmented in DCNM but it's just cosmetic and does not prevent any functionality (CFS, IVR topology distribution ..etc) . I've setup IPFC before and it's the most awkward thing to use, in my environment we use IP based console aggregators that connect to special management networks so if primary mgmt0 interfaces are offline we can always get on the console port.

Good information Swen Jung

Thank you for the attention and the question.

One note first: I don’t recommend using the default VSAN for production; I suggest moving all ports to other VSANs or to the isolated VSAN.

Why I don’t like to have a segmented default VSAN0001:

• Two month ago, doing an ISSU was not possible caused by an fcdomain mismatch detected by the embedded pre-checks… After dissolve the VSAN 1 segmentation upgrading NX-OS was possible
• I recommend using Fabric Manager / DCNM and the embedded tools since years - If VSAN 1 is segmented, I saw lots of scrambled views and misleading warning/error messages
• I prefer to use the default VSAN for doing topology changes, fabric merges, aso – First, I bring up and test new ISLs / port channels using VSAN 1, then I add the required VSANs to the trunk allowed list.

For some reasons, I suggest to follow the Cisco recommendation bellow to deploy an In-Band IPFC management using the default VSAN:

Configuring In-Band Management

The in-band management logical interface is VSAN 1. This management interface uses the Fibre Channel infrastructure to transport IP traffic. An interface for VSAN 1 is created on every switch in the fabric. Each switch should have its VSAN 1 interface configured with an IP address in the same subnetwork. A default route that points to the switch that provides access to the IP network should be configured on every switch in the Fibre Channel fabric.

(Source: Cisco Prime DCNM Installation Guides, http://www.cisco.com/c/en/us/support/cloud-systems-management/prime-data-center-network-manager/products-installation-guides-list.html)

Some use cases for IPFC:

• Last month, a customer was locked out of some MDS-9396S switches caused by Cisco Bug Id CSCuu67700 - M9396S: Mgmt Port connectivity down after duplex parameter change.
  • To avoid from running into similar issues again, the customer is going to deploy IPFC using VSAN 1, too.

• For a worldwide customer, I designed a MDS 9000 I/O Accelerator solution to optimize MirrorView between some data centers. The IOA cluster nodes needs to be able to talk to each other using a always available IP management network.
  • The solution: Using in-band IPFC management and the IOA built-in reliability protocol (LRTP) to detect and recover from ISL failures

• In some deployments, new MDS switches where up and running and could not be configured caused of management access / firewall issues
  • But in-band management could be used to configure the environment; the engineer installing the switch can configure IPFC easily while walking through the setup utility, too.

when I use persistent fcids in this case, will there be an issue with fcid mismatch?

ie if a server in vsan 2 and server in vsan 3 had the same fcid. when I merge them will it cause an issue.