Welcome to this EMC Support Community Ask the Expert conversation. This promises to be a an exciting and informative event for those who manage and build their data centers and want to get some great tips and tricks on how to configure FC routing for best results and a slightly easier life.
Allen Ward, an EMC Elect Member and Founder, has been with the IT department of a large financial services organization for over 15 years now and dedicated to Storage technologies for the last 12+ years. He started out with CLARiiON FC4700s and has worked his way through the CLARiiON line and on to his company's newest VNX7500s with all the bells and whistles. Part way through that period he picked up support for a Symmetrix 8830 and then implemented some new DMX3 arrays into the enterprise. In the last couple of years Allen's Company put in their first VMAX arrays as well. Allen has been heavily involved in the EMC Community Network and EMC Support Forums for several years and holds EMC Proven Professional Expert certifications in both CLARiiON and Symmetrix technologies.
Dynamox, an EMC Elect member and Founder, works for a private research university. In his role as storage administrator he supports nine academic divisions and one non-profit hospital. He's been in the storage industry since 1999 when he first started with Dell PowerVault storage arrays. Currently he supports multiple storage platforms from EMC: Clariion, VNX, Symmetrix (DMX/VMAX), Celerra and Isilon. He is a frequent support forum contributor, and a Support Forum MVP who enjoys helping and learning from his peers in the EMC Support Community Forums.
Rob Koper , an EMC Elect Member and Founder, is working in the IT industry since 1994 and since 2004 working for Open Line Consultancy. He started with Clariion CX300 and DMX-2 and worked with all newer arrays ever since, up to current technologies like VNX 5700 and the larger DMX-4 and VMAX 20k systems. He's mainly involved in managing and migrating data to storage arrays over large Cisco and Brocade SANs that span multiple sites widely spread through the Netherlands. Since 2007 he's an active member on ECN and the Support Forums and he currently holds Proven Professional certifications like Implementation Engineer for VNX, Clariion (expert) and Symmetrix as well as Technology Architect for VNX, Clariion and Symmetrix.
This event begins on April 15th, 2013, and the discussion will be opened then for questions . Allen,Dynamox, and Rob will then be taking your questions then until May 1st. Plus all three experts will record a wrap of this discussion from Vegas @ EMC World 2013 in the exclusive EMC ELECT corner !
Watch this space for further details
This discussion is now open for questions on Fiber Channel Routing. We look forward to a lively interactive and informative discussion.
Regardless of the driving factors behind the need, at some point we are likely to come up against a requirement to allow connectivity between isolated FCSAN islands without actually merging those islands together. You might be attempting to provide connectivity between geographically diverse data centers. You might be looking for a solution for shared infrastructure in a collocated data center (with strict rules around what can and can’t be shared). Or maybe you are running up against the limits of individual FC fabric sizes. The underlying reasons may be technical or political.
The reasons don't matter as much as the need… and the need to provide some level of connectivity without opening the floodgates is a perfect use case for Fibre Channel Routing. Join us for an overview of the technology and a deeper dive into the workings and actual implementation of this solution.
To me the following likely scenarios come to mind when thinking about separate networks: in the LAN world you probably already work with VLANs to isolate certain traffic from traffic in other areas in your network. You can see them as being separate islands, or networks. Sets of switches that don’t touch each other anywhere can be considered the perfect separation. However, in a network where you only need just a few ports it’s much more cost effective to share hardware and using VLANs makes this possible. Traffic that belongs in one VLAN cannot touch traffic in other VLANs.
So far so good, this is what most people will understand and probably are using today. Suppose you have your end users who shouldn’t be able to contact servers through the backup interface and system managers that aren’t supposed to perform their daily tasks using the front LAN, then you probably are already using 3 or more VLANs to keep specific traffic separated from the rest.
Another scenario would be when you want to connect 2 data centers, but you don’t want to stretch a single VLAN from one DC into the other. You’ll have a VLAN in the first DC and another VLAN in the other DC.
Now what if you want traffic to go from a server in the first DC to a server in the other DC? And that’s where it becomes interesting! Since the networks don’t touch each other anywhere, you need to ROUTE traffic from the first network to the other using specific hardware called ROUTERS.
In SAN land it’s no different. I’ve seen many occurences where storage arrays were placed in a separate storage network called a SAN and servers and their HBAs in another network. For example you run a data center with only storage arrays and you are providing storage on demand to customers’ servers. Somehow their HBAs need to be able to reach the storage arrays to send and retrieve data. Here’s where storage routing comes in! Much like routing in the LAN world (think about the internet for example) in data centers storage routing is often used to separate traffic with certain characteristics from other traffic. For example backup traffic from storage array to a tape library which is using Fibre Channel shouldn’t interfere with FC traffic that flows between servers and storage arrays.
So there are a number of scenarios you can think of when you would actually like to start using Fibre Channel Routing.
So how do you feel about routing FC traffic?
It seems we are off to a bit of a slow start so I'm going to throw a question out to my esteemed colleagues about the Cisco side of the world.
With the Brocade hardware I work with, it used to be that you had to buy special Fibre Channel Routing hardware platforms, or a special routing blade for the director class chassis. Now almost all of Brocade's hardware platforms are FCR capable with the addition of a single license to enable all eligible ports on the switch.
How does licensing work on the Cisco hardware?
on the Cisco side you have to have purchase special license for all switches that will participate in IVR topology (inter vsan routing). This license is called ENTERPRISE_PKG and it provides the following features (in addition to IVR)
Advanced Traffic-Engineering Features
The Cisco MDS 9000 Family Enterprise Package includes the following advanced traffic-engineering features:
• Inter-VSAN Routing (IVR): IVR allows selective transfer of data traffic between specific initiators and targets on different virtual SANs (VSANs) without merging VSANs into a single logical fabric. Fibre Channel control traffic does not flow between VSANs, nor can initiators access resources except the ones designated with IVR. In this way, IVR facilitates sharing of resources across VSANs without compromising the VSAN benefits of scalability, reliability, availability, and network security.
IVR also works across WANs over the Fibre Channel Interface Protocol (FCIP). IVR can be used in conjunction with FCIP to create more efficient business-continuity and disaster-recovery solutions. With the introduction of IVR, Cisco has become the first vendor to provide routing capability for Fibre Channel networks in SAN switches.
• Quality of service (QoS): The QoS feature in Cisco MDS 9000 NX-OS Software allows traffic to be classified into four distinct levels for service differentiation. QoS can be applied to help ensure that Fibre Channel data traffic for latency-sensitive applications receives higher priority over throughput-intensive applications such as data warehousing.
– Zone-based QoS is included in the Cisco MDS 9000 Family Enterprise Package and complements the standard QoS data-traffic classification by VSAN ID, N-port worldwide name (WWN), and Fibre Channel identifier (FC-ID). Zone-based QoS helps simplify configuration and administration by using the familiar zoning concept. QoS can also be configured per VSAN or be policy or class based.
• Extended credits: The extended credits feature allows up to 3500 credits to be assigned to a single Fibre Channel port within a group of four Fibre Channel ports. Adding credits increases distances for Fibre Channel SAN extension. This feature is available only on the Cisco MDS 9000 Family Multiprotocol Services Module (MSM) and Cisco MDS 9216i Multilayer Fabric Switch.
• Small Computer System Inteface (SCSI) flow statistics: Logical unit number (LUN)-level SCSI flow statistics are collected for any combination of initiator and target. The scope of these statistics includes read, write, and error statistics. This feature is available only on Cisco MDS 9000 Storage Services Modules and MDS 9000 Advanced Services Modules.
Not all switches support Inter VSAN Routing, the models that do support it are: MDS 9148, MDS 9222i, MDS 9509 and MDS 9513
So there are two interesting differences in licensing right there. If I'm interpreting your note correctly, IVR requires all switches involved in the routing to be licensed and to get the license requires licensing a number of other features as well in a package. Mind you the other features in the package all (or mostly) appear to be good things to have, and some of them especially good if you are dealing with routing over distance.
In the Brocade world the Integrated Routing license is a separate licenses and then most of the other equivalent features to those you list can be acquired as an "Enterprise Bundle" as well.
The other major (bigger) difference I see here is that with Brocade FCR you only need to license one of the switches involved in the routing. If you go back to my blog and look at options you will see that to connect simple limited routing between two fabrics you can just apply an IR license on a single switch in one of the fabrics and use that one to make the IFL to a switch in the other fabric. Or you could license a single switch to act as the backbone for connectivity between multiple fabrics. The edge fabrics with the devices that need to communicate don't have to be licensed for FCR.
BTW - (and for the official record) I'm not attempting to highlight the differences in an effort to show one or the other as better. This is just to help the understanding of both and the options available as well as any considerations that may sway a decision one way or another.
let me provide a little more details when i say "all" switches need to be licenses. I guess i can refer to them as "border" switches ..they are the ones that connect two distinct fabrics together. In my picture below it would be 9513 and 9222i would be the ones that would require IVR license. The 9124e and 9134 does not support nor do not require IVR licenses. As long as the both switches (for example 9222i and 9134) share the same VSAN 110 ..only the 9222i needs to be licensed.
Just for the sake of clarity, the concept of VSANs don't exist in the Brocade world (or at least not by that name). Are you referring to a specific fabric, or is a VSAN another kind of construct specific to Cisco deployments?
I think you can compare VSANs with Brocade's ability to use fabric ids. If you don't want 2 switches to merge you can give them different fabric ids, but VSANs are more like VLANs, so within a single physical switch more VSANs can exist. No merging will take place and you need an FC Router to route traffic from one VSAN to another VSAN.
In Brocade land I think this can be done with fabric ids in virtual switches, but I don't think it's entirely the same.
If you have 2 Brocades connected to each other by an ISL and each switch has 2 virtual switches defined, can you merge one from each physical switch with one of the other one?