Unsolved
This post is more than 5 years old
19 Posts
0
3709
ISL creation across DWDM link
I need a sanity check for the following process to create ISLs for a dedicated cross-site replication VSAN. I have to provide a process to the implementation guys, but do not have access to test switches to test the process. I really appreciate it if you could cast your eye over this, and let me know whether it's correct or not. Here are the key facts:
- MDS9513's
- ports are on 24-port 8G line card, and the only members of the replication VSAN (once the replicaiton devices are in place they will also be members)
- to be connected over a 4G DWDM link
- only dedicated replicaiton VSAN to merge
- ISL to carry dedciated replication VSAN traffic only, so trunking off for these ports
- configure trunk allow list in case trunking gets turned back on for these port
- VSAN1 must NOT merge - it is being used for host attach (this is will cleared out, but not before the ISL is created across sites)
1. Create dedicated replication VSAN on each switch;
2. Ensure the Domain ID for each switch & VSAN are unique;
3. Ensure the ISL port is admin down, and then modify settings in following order:
- Change VSAN to the Replication VSAN;
- Change port mode to E;
- Change trunk allowlist to include the Replication VSAN only;
- Disable trunking; (is the allowlist persistent? will it exist is trunking was to be enable again?)
- Change rate-mode from shared to dedicated;
- Change speed from auto to 4000;
- Add description
4. Cable switches to DWDM mux;
5. Bring the ports online;
6. Check interfaces for errors
7. Check for VSAN merge errors
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 29th, 2013 07:00
i would be concerned about VSAN0001 because if i look at my switches where i have connected 9513 to a 9222i, VSAN0001 became segmented which makes sense because i am not trunking it but what i also see is that VSAN0001 on 9222i is operationally down. I know i did not take it down manually so it must have taken it down by itself when i first connected these two switches together.
Safe to assume you are not licensed for IVR so you are not going with transit VSAN to carry your replication VSAN traffic ?
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 29th, 2013 08:00
forgot to mention that i leave my ports that go to my DWDM gear in VSAN1, you can't delete VSAN1 so you can't take my replication down (if somebody accidentally deleted my transit vsan)
ScottHolmanColt
19 Posts
0
January 29th, 2013 08:00
Well, the switches do have the enterprise license, so I could use IVR/transit VSAN. However, the only cross site requirement is to connect the RecoverPoint deivces. There's no intension to allow hosts to connect to storage across DC boundaries. Also, I like to keep it simple, as I do not have much faith in the support team.
VSAN1 segmentation is interesting. Have you merged VSAN1 across your tranit VSAN? If not has VSAN1 segmented on either switch? Also, if we assume VSAN1 to be clear of devices does it matter?
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 29th, 2013 08:00
even though i do not have any devices in VSAN1 i did not want to "trunk allow" VSAN1 over my transit VSAN just to get rid of segmentation. In one of my early Cisco MDS classes i was told that VSAN1 carries some information that you don't want to merge with VSAN1 of another switch. So i have always had VSAN1 segmented, looks ugly in Fabric Manager but it does not bother me.
I have local replication requirements (RDF between DMX4 and VMAX, same datacenter) so i did not want to trunk that VSAN to my DR site, hence using IVR.
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 29th, 2013 13:00
better feedback from Cisco community ?
ScottHolmanColt
19 Posts
0
January 30th, 2013 00:00
ha, ha. I've seen you on there, so knew it wouldn't be long before you'd pick that one up. I thought a bit more exposure wouldn't hurt.
anyway ... in answer to your question ... no ... I've more confused than when I started. My question now is ... what is the best practice for VSAN1 ... to merge or not to merge across an ISL. Is it a good or bad idea? I can't find official Cisco doc that states best practice for this.
This has got me thinking. I know that best practice dictates that VSAN1 should not be used for prod traffic. Some say it's for staging (cisco doc), some avoid using (cisco docs), and some take it offline (community), but I can't find why this is best practice. I've blindly adhered to this for years without really understanding why.
Another question: if you have the DWDM ports in VSAN1 do you not have to create an ISL between those ports requiring you to merge VSAN1? or have I just completely exposed by lack of understanding of tranist VSANs.
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 30th, 2013 04:00
Scott,
i have two 9222s in my lab that i could use to test this out, and see if i experience the same issue as i have in my production fabric. What code are you running on your switches and i will upgrade these 9222is to the same one.
I apologize if i repeat what you already know but let's assume this is my requirements, i need a host in vsan 10 be able to talk to storage array in vsan 20. Merging fabrics here is not even an option so we are going to use IVR. To establish connectivity i perform these steps:
1) connect my single mode fiber from DWDM gear to long-wave SFP on 9513, port is configured for VSAN1
2) connect my single mode fiber from DWDM gear to long-wave SFP on 9222i, port is configured for VSAN1
3) create VSAN 30 on 9513 and 9222i, it has no members, it has not zonesets ..nada
4) create IVR topology and activate it
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:f0:16:00 vsan-ranges 10,30 <<- 9513 N
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:f0:17:00 vsan-ranges 20,30 <<- 9222i
5) trunk allow VSAN 30 on the ports we connected to in step 1.
6) at this poing you can IVR zone devices from both fabrics.
ScottHolmanColt
19 Posts
0
January 30th, 2013 06:00
aaarrgghhh. I just lost my post for the third time. Fortunately I took a copy of the second attempt.
Ok. So, I'll paraphrase the lost post. First of all ... thanks for the offer to test. My test lab funding was canned again. grrr.
So, if I use your example I want to simply stretch VSAN 10 across sites using the ISL. VSAN 10 will contain the DWDM connected ports, and the RecoverPoint ports only, so that the RPAs can communicate. no requirement for IVR, etc.
If I apply a trunk allow list to include VSAN 10 only excluding VSAN1 then VSAN1 will not merge. At the moment I do not want VSAN1 to merge as one site has a bunch of UCS environments attached to it.
There a post on the cisco forums (one to which you've contributed Mr D) that has a Cisco employee stating that VSAN1 should be merged across an ISL, as it propagates fabric info. Of course it's assumed that it's free of prod devices. The problem is that when pressed the Cisco chap didn't offer up official documentation that would corroborate this.
https://supportforums.cisco.com/message/3401997#3401997
What worried me is possibility of one of the VSAN1 being forced operationally down when VSAN1 becomes segmented, as you detailed in an earlier post. Are you able test whether this does actually happens?
I'm leaning towards making VSAN1 cleaning a prerequisite to the ISL creation and then merging VSANs 1 & 10. I'll ask my TC for advice from an EMC MDS specialist too. I'll let you know the response.
cheers
scott
ScottHolmanColt
19 Posts
0
January 30th, 2013 06:00
forgot to mention one side has 5.0(1a) & the other 4.2 (something)
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 30th, 2013 09:00
upgrading switches right now, will let you know how it goes
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
2
January 30th, 2013 14:00
Scott,
this is what i have tested so far and my results:
9222i - 5.0(1a)
9222i - 4.2(7e)
1) one connected to another one using single mode cable, VSAN1 is down on both switches, ports are in VSAN1.
2) configured VSAN 10 on both switches, and set ports to mode E, at this point VSAN 1 came up on both switches, VSAN 10 is still down (makes sense because it's not being trunked yet)
3) configure ports to trunk mode "on" and trunked allowed vsan 10. At this point VSAN 10 came up and VSAN 1 on both switches went down.
Let me know if you want me to test any other scenarios.
ScottHolmanColt
19 Posts
0
January 31st, 2013 01:00
Thanks very much Mr D. The joys of having your own lab.
Presumably VSAN1 would remain up if it had other active ports. Also, is it fair to assume that VSAN1 would remain up if added to the allow list (I think so)
I'm struggling to find a collectively agreed answer to whether VSAN1 should be merged across ISLs. Some, like yourself, say not, some say yes as it some features use it for config & management traffic. It would be nice to get an official Cisco response other than a few Cisco employees making statements it in forums.
All in all it does not seem to matter a great deal. You have an environment that does not trunk VSAN1, and you do not have issues other than VSAN1 appearing as segmented in FM, and that's not really a problem just a personal OCD tidiness issue for which I need therapy.
scott
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
0
January 31st, 2013 05:00
unfortunately i don't have any hosts that i could add to VSAN1 and see if it remains online. You are correct, the minute i add VSAN1 to trunk allow list, it comes up ..but at this point you are merging the two which you were trying to avoid. I like your idea of migrating these hosts/arrays to another VSAN before you establish connectivity with another site. I know it's very intrusive and will need downtime but it will help them going forward and help you deal with your OCD
. I will try to get a host and drop it into VSAN1 and repeat the same test.
dynamox
1 Rookie
1 Rookie
•
20.4K Posts
1
February 1st, 2013 11:00
Scott,
i got a host in my lab and i repeated the same test, i think you will like the results. As you can see from the results, VSAN1 did not go down and the only reason i can think of is because i actually have systems in VSAN1 where in my production fabrics i don't have any systems that are in VSAN1.
ScottHolmanColt
19 Posts
0
February 5th, 2013 03:00
Thanks for your help on this. It's been great to have a sound board.
You're correct I have been trying to avoid merging VSAN1. However, I've now made cleansing VSAN1 of devices a prerequisite to the distance extension, so that VSAN1 can be merged.
My reasoning is the that after discussing this with a few EMC MDS specialists, and colleagues they've all agreed that merging an empty vsan1 is preferred, and considered best practice. There are also a few Cisco employee's in the Cisco forums that say the same.
That stated I've concluded that under certain circumstances, and in some environments having VSAN1 segmented is also valid, as long as you're happy with managing it that way, AND you're not using MDS features that require VSAN1 to communicate, e.g. like CFS. As I've not been able to find an official Cisco explanation of the features requiring VSAN1 I'm now of the opinion that VSAN1 should be empty and merged to avoid an unforeseen issues should the more advanced features be enabled.
Thanks again for your time and help MR D, and use of your lab .... I really must get one of those!