Ask the EMC Elect Experts about Fibre Channel Routing

This discussion is now open for questions on Fiber Channel Routing. We look forward to a lively interactive and informative discussion.

Regards,

Mark

I know RRR & Dynamox are going to be jumping in with some intro material soon, but in the mean time, if you don't follow my blog, check out this recent post for some background on Fibre Channel Routing from a Brocade perspective.

To me the following likely scenarios come to mind when thinking about separate networks: in the LAN world you probably already work with VLANs to isolate certain traffic from traffic in other areas in your network. You can see them as being separate islands, or networks. Sets of switches that don’t touch each other anywhere can be considered the perfect separation. However, in a network where you only need just a few ports it’s much more cost effective to share hardware and using VLANs makes this possible. Traffic that belongs in one VLAN cannot touch traffic in other VLANs.

So far so good, this is what most people will understand and probably are using today. Suppose you have your end users who shouldn’t be able to contact servers through the backup interface and system managers that aren’t supposed to perform their daily tasks using the front LAN, then you probably are already using 3 or more VLANs to keep specific traffic separated from the rest.

Another scenario would be when you want to connect 2 data centers, but you don’t want to stretch a single VLAN from one DC into the other. You’ll have a VLAN in the first DC and another VLAN in the other DC.

Now what if you want traffic to go from a server in the first DC to a server in the other DC? And that’s where it becomes interesting! Since the networks don’t touch each other anywhere, you need to ROUTE traffic from the first network to the other using specific hardware called ROUTERS.

In SAN land it’s no different. I’ve seen many occurences where storage arrays were placed in a separate storage network called a SAN and servers and their HBAs in another network. For example you run a data center with only storage arrays and you are providing storage on demand to customers’ servers. Somehow their HBAs need to be able to reach the storage arrays to send and retrieve data. Here’s where storage routing comes in! Much like routing in the LAN world (think about the internet for example) in data centers storage routing is often used to separate traffic with certain characteristics from other traffic. For example backup traffic from storage array to a tape library which is using Fibre Channel shouldn’t interfere with FC traffic that flows between servers and storage arrays.

So there are a number of scenarios you can think of when you would actually like to start using Fibre Channel Routing.

So how do you feel about routing FC traffic?

It seems we are off to a bit of a slow start so I'm going to throw a question out to my esteemed colleagues about the Cisco side of the world.

With the Brocade hardware I work with, it used to be that you had to buy special Fibre Channel Routing hardware platforms, or a special routing blade for the director class chassis. Now almost all of Brocade's hardware platforms are FCR capable with the addition of a single license to enable all eligible ports on the switch.

How does licensing work on the Cisco hardware?

on the Cisco side you have to have purchase special license for all switches that will participate in IVR topology (inter vsan routing). This license is called ENTERPRISE_PKG and it provides the following features (in addition to IVR)

Advanced Traffic-Engineering Features

The Cisco MDS 9000 Family Enterprise Package includes the following advanced traffic-engineering features:

• Inter-VSAN Routing (IVR): IVR allows selective transfer of data traffic between specific initiators and targets on different virtual SANs (VSANs) without merging VSANs into a single logical fabric. Fibre Channel control traffic does not flow between VSANs, nor can initiators access resources except the ones designated with IVR. In this way, IVR facilitates sharing of resources across VSANs without compromising the VSAN benefits of scalability, reliability, availability, and network security.

IVR also works across WANs over the Fibre Channel Interface Protocol (FCIP). IVR can be used in conjunction with FCIP to create more efficient business-continuity and disaster-recovery solutions. With the introduction of IVR, Cisco has become the first vendor to provide routing capability for Fibre Channel networks in SAN switches.

• Quality of service (QoS): The QoS feature in Cisco MDS 9000 NX-OS Software allows traffic to be classified into four distinct levels for service differentiation. QoS can be applied to help ensure that Fibre Channel data traffic for latency-sensitive applications receives higher priority over throughput-intensive applications such as data warehousing.

– Zone-based QoS is included in the Cisco MDS 9000 Family Enterprise Package and complements the standard QoS data-traffic classification by VSAN ID, N-port worldwide name (WWN), and Fibre Channel identifier (FC-ID). Zone-based QoS helps simplify configuration and administration by using the familiar zoning concept. QoS can also be configured per VSAN or be policy or class based.

• Extended credits: The extended credits feature allows up to 3500 credits to be assigned to a single Fibre Channel port within a group of four Fibre Channel ports. Adding credits increases distances for Fibre Channel SAN extension. This feature is available only on the Cisco MDS 9000 Family Multiprotocol Services Module (MSM) and Cisco MDS 9216i Multilayer Fabric Switch.

• Small Computer System Inteface (SCSI) flow statistics: Logical unit number (LUN)-level SCSI flow statistics are collected for any combination of initiator and target. The scope of these statistics includes read, write, and error statistics. This feature is available only on Cisco MDS 9000 Storage Services Modules and MDS 9000 Advanced Services Modules.

Not all switches support Inter VSAN Routing, the models that do support it are: MDS 9148, MDS 9222i, MDS 9509 and MDS 9513

let me provide a little more details when i say "all" switches need to be licenses. I guess i can refer to them as "border" switches ..they are the ones that connect two distinct fabrics together. In my picture below it would be 9513 and 9222i would be the ones that would require IVR license. The 9124e and 9134 does not support nor do not require IVR licenses. As long as the both switches (for example 9222i and 9134) share the same VSAN 110 ..only the 9222i needs to be licensed.

So there are two interesting differences in licensing right there. If I'm interpreting your note correctly, IVR requires all switches involved in the routing to be licensed and to get the license requires licensing a number of other features as well in a package. Mind you the other features in the package all (or mostly) appear to be good things to have, and some of them especially good if you are dealing with routing over distance.

In the Brocade world the Integrated Routing license is a separate licenses and then most of the other equivalent features to those you list can be acquired as an "Enterprise Bundle" as well.

The other major (bigger) difference I see here is that with Brocade FCR you only need to license one of the switches involved in the routing. If you go back to my blog and look at options you will see that to connect simple limited routing between two fabrics you can just apply an IR license on a single switch in one of the fabrics and use that one to make the IFL to a switch in the other fabric. Or you could license a single switch to act as the backbone for connectivity between multiple fabrics. The edge fabrics with the devices that need to communicate don't have to be licensed for FCR.

BTW - (and for the official record) I'm not attempting to highlight the differences in an effort to show one or the other as better. This is just to help the understanding of both and the options available as well as any considerations that may sway a decision one way or another.

Ok, so the 9148 is a layer 3 switch? I didn't know that. I know that in the past the 9124 and 9134 were falsely mentioned as being capable of routing. If the 9148 is capable of routing, then this would make this switch perfect for a lot of routing within the same date center, right? We wouldn't have to buy an expensive 9222i anymore.

The 9222i Does have more buffer to buffer credits and the FCIP capability as well and it has the extra slot, so you can expand this switch.

Can you give an example which switch do and which don't have the routing capability?

The DS200 and DS300 type switches aren't capable and the 7500 and DCXs are, but I'm sure I'm probably not mentioning the numbers correctly.

And how does Brocade compare to the features Dynamox just mentioned? I know that IVR compares to LSANs in the Brocade world and uses "EX" ports instead of "E" ports or is there more to it.

I'll take this one in two parts RRR, starting with the question of which Brocade platforms are capable of FCR.

Pretty much all the mainstream FC switches in Brocade's lineup are capable of FCR including the 5100, 5300, 6510, 7800, the DCX and 8510 series of backbone directors and a few other "miscellaneous" switches in the lineup. Generally the switches that are considered to be "entry level" (e.g. 300, 6505, etc.) do not offer this functionality. A few of the options (like the 5100, 6510, and 7800) are a bit better for certain long distance extended connection due to the way they handle sharing and allocation of BB credits, but if you are doing FCR within a data center or over shorter distances that doesn't matter as much.

I'm trying to remember the details on the 200, and all that comes to mind is the integrated switching module included in some blade chassis. If that is the one you are talking about I believe it falls outside the scope of switches that can handle FCR, but we would have to dig into the Brocade documentation to be sure. Is thata even a support product any more?

Just for the sake of clarity, the concept of VSANs don't exist in the Brocade world (or at least not by that name). Are you referring to a specific fabric, or is a VSAN another kind of construct specific to Cisco deployments?

For the other part of your question RRR, the other Cisco features Dynamox mentioned all have equivalents in the Brocade world except for the low level SCSI flow control stats. There are some alerting features that you can set to trigger on certain low level SCSI frames, but it doesn't sound like it is quite as comprehensive. There are some other features you gain with the Enterprise Bundle of licenses, but they aren't really directly related to FCR. If we have time later we can dig into them if this discussion doesn't pick up a bit, but I don't want to dive too deep into things outside the defined scope just yet.

I think the 300 is the new 200, both entry level.

I think you can compare VSANs with Brocade's ability to use fabric ids. If you don't want 2 switches to merge you can give them different fabric ids, but VSANs are more like VLANs, so within a single physical switch more VSANs can exist. No merging will take place and you need an FC Router to route traffic from one VSAN to another VSAN.

In Brocade land I think this can be done with fabric ids in virtual switches, but I don't think it's entirely the same.

If you have 2 Brocades connected to each other by an ISL and each switch has 2 virtual switches defined, can you merge one from each physical switch with one of the other one?