Ask the EMC Elect Experts about Fibre Channel Routing

Thanks Rob. I had to go back and dig a bit, but I did find that the 200 was the entry level prior to the 300 (and now the 6505). It definitely did not offer routing capability any more than its ancestors from this century :-)

Allen Ward wrote: . Realistically, if you were going to ISL them together internal to the physical chassis why bother making them separate virtual switches in the first place?

you could have a dedicated virtual switch for tape only connectivity where you would put HBAs of your backups servers, tape drives/FC ports of your VTLs.  Now let's say you have a two node Oracle cluster that needs to be backed up LAN-Free. You could drop another HBA and add it to the tape virtual switch (best option) or you could present tape resource just to this server from the other virtual switch using FCR.

Hmmm... there does seem to be more of a difference here than the surface revealed. In the Brocade hardware (assuming first that you are only dealing with physical chassis as a whole) there is no real concept of setting a fabric ID. If you don't want two fabrics to merge you just don't ISL them together :-) If you want them to allow limited connectivity between a subset of devices without merging you use FCR. With FCR added to the mix there is a concept of FIDs, but it only happens when you define an EX port for an IFL to another switch.

Now, if you add in the switch virtualization feature things still don't change that much. I haven't worked a lot with the virtualization features because we don't have a real use case for it. But to your specific question: If I take a pair of 5300s and configure them each with the first 40 ports defined to one virtual switch and the last 40 ports defined to another virtual switch, then connect ports 0 & 1 of each together, the virtual switches using those ports will ISL together (assuming that all the other rules are met to allow an ISL). I can't remember, but I seem to remember a way of getting the virtual switches within a single chassis to talk to each other, but I seem to remember it involving FCR as opposed to a direct ISL. Realistically, if you were going to ISL them together internal to the physical chassis why bother making them separate virtual switches in the first place?

I totally agree that routing is a potential solution to this use case if you have a valid reason for not doing it the "best way". My point was more about ISL than IFL though and was more rhetorical.

It sure does

http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps4358/product_data_sheet09186a00801cc917.html

Hi,

I think that IVR accross the FCIP Link is included with the SAN Extension  ( FCIP) License.  This would allow you the ability to connect 2 remote SAN Island without merging, or buying an additional license

I was thinking about 2 v-switches in each physical one indeed, but when you ISL them, in which v-switch is the ISL? Does that even matter? And what if I only want 1 of the 2 v-switches in each physical switch to merge?

So I have 2 physical switches (black) and each one devided into 2 virtual ones for some reason (red and green). Now I want the 2 red ones merged, but not the green ones. In Cisco I'd allow the red VSAN to use the portchannel and not the green VSAN(s), but how can I do this in Brocade? I know this is not true routing yet, but it sure is important!

Hi,

FCIP interfaces are simply ISL’s as far as Fibre Channel is concerned. When configuring them an option is to only allow specific VSANs to use them. Cisco allows the editing of a “trunk-allowed list” for any ISL. This determines which VSAN(s) will merge across the ISL. This can be used to only allow a designated VSAN to merge across the IP connection. So local traffic is unaffected, for example if the FCIP connections “bounce” on and off.

The use of IVR and a transit VSAN can further expand the connectivity options. That probably was the thinking behind why Cisco allows the use of IVR across the FCIP interfaces without an Enterprise license. Of course you do need a SAN Extension license to use FCIP

Paul,

I'm 100% certain that when connecting 2 switches together using FCIP the 2 fabrics will merge! If I remember correctly FCIP will simply tunnel FC using tcp/ip and it can be seen as an ISL, so both ends will in fact merge.

iFCP, which I've never seen in my life just yet, does what you're saying: connect the two, but no merge will take place.

Can anyone confirm this?

So thus far we talked about routing to other (virtual) networks like VSANs , virtual switches or other locations. But in the LAN world we have something called "spanning tree" if for some reason a series of switches are connected together so a loop or multiple loops are formed. Each switch will calculate the best route to any destination so traffic will be delivered as efficient as possible. If for some reason this loop is disrupted the calculation needs to be done all over again to match the new topology and you will notice a thing called spanning tree (or rapid spanning tree) which is disruptive for your environment.

So how does this work in the SAN world? Can we create loops or full meshes and what happens when the topology is disturbed when a switch fails (hardly happens) or someone pulls out the wrong cable (much more likely)?

The answer is FSPF = Fabric Shortest Path First

Fabric Shortest Path First
According to the FC-SW-2 standard, Fabric Shortest Path First (FSPF) is a link state path selection protocol. The concepts used in FSPF were first proposed by Brocade and have since been incorporated into the FC-SW-2 standard. Since then, it has been adopted by most, if not all, manufacturers.

What is FSPF?
FSPF keeps track of the links on all switches in the fabric and associates a cost with each link. At the time of writing, the cost is always calculated as being directly proportional to the number of hops. The protocol computes paths from a switch to all the other switches in the fabric by adding the cost of all links traversed by the path, and choosing the path that minimizes the cost. For example in the figure below, if we need to connect a port in switch A to a port in switch D, it will take the ISL from A to D.

The other possible paths are shown below.


It will not go from A to B to D, neither from A to C to D. This is because FSPF is currently based on the hop count cost.

How does FSPF work?
The collection of link states (including cost) of all switches in a fabric constitutes the topology database (or link state database). The topology database is kept in all switches in the fabric and they are maintained and synchronized to each other. There is an initial database synchronization, and an update mechanism. The initial database synchronization is used when a switch is initialized, or when an ISL comes up. The update mechanism is used when there is a link state change, for example, an ISL going down or coming up, and on a periodic basis. This ensures consistency among all switches in the fabric.

How does FSPF help?
In the situation where there are multiple routes, FSPF will ensure that the route that is used is the one with the lowest number of hops. If all the hops:

• Have the same latency
• Operate at the same speed
• Have no congestion

Then FSPF will ensure that the frames get to their destinations by the fastest route.

What happens when there is more than one shortest path?

If we look again at the example in the first figure, and we imagine that the link from A to D goes down, switch A will now have four routes to reach D:

• A-B-D
• A-C-D
• A-B-C-D
• A-C-B-D

A-B-D and A-C-D will be selected because they are the equal shortest paths based on the hop count cost. The update mechanism ensures that switches B and C will also have their databases updated with the new routing information. So, which of the two routes will be used? The answer is that the decision of which way to send a frame is up to the manufacturer of each switch. In our case, Switch B and Switch C will send frames directly to Switch D. The firmware in Switch A will make a decision about which way to send frames to Switch D, either via Switch B or Switch C. The way that this decision is made is by a round robin algorithm based on the order of connection. Let us consider the situation illustrated in the figure below:

There are three servers A, B and C which all need to communicate with the storage devices D, E and F respectively (We are assuming that there is no zoning or trunking enabled, and that all of the links are operating at the same bandwidth. Let us assume that the three servers connect in the order A then B then C. Server A will be given a route from the upper switch to the lower switch. For the sake of this example, let us assume that it is via ISL1. The second server, Server B in the example will be assigned a route via ISL2. and the Server C will have a route via ISL1. This will have the result of sharing the load between the two switches over the two ISLs. We can see that some traffic will flow via each of the ISLs, but we must stress that this is not the same as load balancing; this implements load sharing but not load balancing.

(source: IBM)

Ah ok, that's what you meant. Yes, by using the allowed list you can either allow or disallow certain VSANs from using the ISL.

I'm confused by the question, but I think it must relate to the way "virtual switches" are defined within a physical chassis. In the Brocade world I define specific physical ports on the physical chassis to be associated with each virtual switch. So in your example, the red virtual switch in each chassis might have ports 0-39 and the green could be ports 40-79. So if you ISL using ports 0 & 1 then you are clearly merging the red virtual switches into a fabric and the green ones don't know anything about each other. If you wanted something different you would have to configure something special to allow for it. I do remember when I was digging into the virtualization features a while back that there were ways to share virtual ISLs, but by default that wouldn't happen.

Maybe things have changed now, but again... this is not a feature that I deal with today so I don't have a deep understanding of it. Anyone out there working with switch virtualization in the FOS?

In the Cisco world it's very common to make ISL available to many VSANs at the same time. The ISLs themselves are most likely part of the default VSAN 1 and you ALLOW certain VSANs to cross that ISL. So going back tto my simple drawing, an ISL doesn't belong to the red or the green virtual switch, but I'd simply ALLOW only the red VSAN / virtual switch / fabric to use that link.

I do remember when I was digging into the virtualization features a while back that there were ways to share virtual ISLs, but by default that wouldn't happen.

And that's what I'm talking about actually.

It seems the discussion now comes down to naming differences between Brocade and Cisco, but I guess that's because the two are the two major players in the SAN world nowadays.

Anyone out there working with switch virtualization in the FOS?

One example would be where you want to use an isolated fabric for your replication traffic, another one for connecting to a certain customer that you don't want to see the contents of your own fabric and then the fabric where you put in your hosts and FA ports.

But let's wait a bit to see if anyone "out there" is using it. I know there are companies that are doing it, but it's nice to hear from them directly.

Wow, great discussion! I am looking forward to your wrap up video at EMC World on this. The Whiteboard is going to be a busy place:)