Ask the Expert: SAN (Connectrix), FC Connectivity Recommendations and Best Practices

Good point Allen, I should've clarified that as well in order to avoid any possible confusion.

Thanks for taking time in writing!

Felipe

Good morning @RRR,

In response to your query,

Do you know of or have any experience with running Cisco port channels over CWDM (or DWDM) at for example 60 km at the highest speed possible? I have two data centers, about 60 km apart and we're running the port channel over CWDM at 4 Gbps speed, but our fabrics on both sides of that dark fiber are 8 Gbps. Obviously I want to match the speed on the long distance link to whatever we're using on both sides, but it seems Cisco doesn't have SFP's at 8 Gbps for CWDM that are able to cross 60 km. What would you advice? Doubling the number of links is not a valid answer since each of the 2 ISLs is still running at 4 Gbps and the frames won't cross the distance at 8 Gbps speed .

With the described scenario, I suppose that 4G is the best you can get for a CWDM implementation. For 8G, you would have to use an active WDM technology that guarantees 8G over that distance. As you mentioned, Cisco 8G CWDM SFPs, supports up to 25 Km and using them wouldn't be a valid option.

Rgds,

Felipe

Hi Felipe,

I would like to know what the different options are when configuring an extended ISL on Brocade. There are various parameters to configure (vc_link_init, R_RDY, LD/LS) and it is not easy to differentiate between them.

Thanks.

Moss

That's just too bad. I found a vendor in China that supposedly has Cisco supported SFP's that can support up to to 80 km, but I wanted to know if anyone has any experience with exotic brands that do so. I know Smart Optics is a commonly used brand, but even those cannot cross 60 km using 8 Gb technology on CWDM.

I've got another one for you: suppose I have 2 sites with a routing 9222i on each side of the dark fiber and I want to upgrade these machines with new 9250is. I would add them to the existing fabric first so it "receives" the current zoneset configuration, but I want the 9250 to replace the 9222. So at some point I need to add the new 9250 to the IVR topology and make the 9222 "go away". Do you have any tips and tricks for this scenario?

Good morning Moss,

Great question, thanks for asking!

In a Brocade switch, there are 3 main parameters to configure during an extended ISL setup:

SWITCH:admin> portcfgshow 7/7

Area Number:              71

Speed Level:              4G <-- It's recommended to set the speed manually, specially on ISLs

Fill Word:                0(Idle-Idle)

AL_PA Offset 13:          OFF

Trunk Port                ON

Long Distance             OFF

VC Link Init              OFF

Locked L_Port             OFF

Locked G_Port             OFF

Disabled E_Port           OFF

Locked E_Port             OFF

ISL R_RDY Mode            OFF

RSCN Suppressed           OFF

Persistent Disable        OFF

LOS TOV enable            OFF

NPIV capability           ON

QOS E_Port                OFF

Port Auto Disable:        OFF

Mirror Port               OFF

F_Port Buffers            OFF

Fault Delay:              0(R_A_TOV)

NPIV PP Limit:            126

• Long Distance mode:
  • L0: standard mode.Specify L0 to configure the port as a regular port.
  • LE: Specify LE mode to configure an E_Ports distance greater than 5 Km and up to 10 Km. This mode does not require the Extended Fabric license.
  • LD: Dynamic Long distance mode. The switch automatically estimates the distance of the link and assigns BB Credits for that distance (for 2148 Bytes frames)
  • LS: Static Long distance mode. You have to specify the distance of the link and the switch will assign BB credits for that distance. With FOS 7 you can directly set the number of BB credits to use.

• VC Link Init:
  • OFF (0): use primitive IDLE as the fillword.
  • ON (1): use primitive ARB as the fillword.

• R_RDY mode:
  • OFF (0): use primitive VC_RDY to free up a Buffer-to-Buffer credit.
  • ON (1): use primitive R_RDY to free up a Buffer-to-Buffer credit. This primitive is the one defined in the FC standard. It is the one used on Cisco switchs.

Brocade recommends to enable VC_LINK_INIT and disable R_RDY mode, but you can configure them at your convenience as some times the WDM/TDM array behind the ISL need to have a special setup.

SWITCH:admin> portcfglongdistance 7/7

Usage: portCfgLongDistance [SlotNumber/]PortNumber       Distance_Level

distance_level:         L0  : normal

                        LE  : <= 10km

                        LD  : auto

                        LS  : static

vc_trans_link_init:     0   normal

                        1   VC translation

desired_distance:       in Kilo-meters

                        valid for LD or LS level only

Please enter the long distance level -- LS

Please enter the vc translation link init -- 1

Please enter the desired distance (in Km) -- 100

Reserved Buffers =        206

In order to make it work, both ends need to be configured with the same setup. But there is an exception, R_RDY mode, when you enable R_RDY mode on one of the ends, no configuration check will be performed and the ISL will get established using R_RDY, and VC_LINK_INIT will automatically be disabled no matter what it was put in the port configuration.

SWITCH:admint> portcfgislmode 7/7 1

ISL R_RDY Mode is enabled for port 71. Please make sure the PID

formats are consistent across the entire fabric.

Best regards,

Felipe

I was wondering since last month we had some issues with an IVR license that accidently ran during 120 days on a switch we didn't know it was running on and the topology changed by itself and when the license ended IVR stopped working, so before I'm going to replace the switches this time, I want to be sure of the steps that need to be taken. From what I've seen it's not that hard at all, but any tips and tricks are welcome.

i can't think of anything exotic that needs to be done:

1) enable full zone distribution on 9250i (hopefully it's already enabled on 9222is

2) add 9250i to ivr topology

3) switch dark fiber links to 9250i when ready

Hi there,

IVR has been around for a little while and  it its a pretty solid feature and, as dynamox  said, no special considerations need to be taken into account.

When it comes to the 8G CWDM SFP query, I've never seen an environment using CWDM working with such distance at 8G and, call me conservative but due to this fact, I don't recommend you to be the first one to check how it works (that's the reason why you're asking I suppose). This Ask the Expert event is about to finish and not many comments may be posted in the near future, but I hope that in the Infrastructure Connectivity Support Forumsomeone may share his experience with this technology.

Felipe

I hope so too. I posted my question there as well.

Hi Felipe,

Question is on right number of paths to a host(RHEL) from VMAX zoning.

For VMAX, we have two engines and each engine has got 2 directors(A and B).  Each engine has 2 directors(odd and even) We normally are zoning servers to two directors (one director from engine-1 and second director from engine-2).   So a total of 2 zones are configured from 2 hbas to 2 targets going through 2 san fabrics.

Checking if this is recommended zoning ? we have 1 initiator and 1 target zoned. Also, ONLY 2 targets making 2 paths to Luns.

FYI, we have connected total 16 ports from VMAX to SAN fabric.

V-Max Best Practice FA connections

Hi HARI.SIRI ,

The basic best practices for Symmetrix have been gathered in KB 174725 and according to this KB:

• Each host should be configured to a minimum of two FA's (SAFs), 4 is recommended.
• Connect at least two HBAs across redundant fabrics for high availability.

Best regards,

Felipe

This Ask the Expert session has concluded. We'd like to thank everyone who participated, but special thanks to our SME Felipe Guijarro  who went out of his comfort zone to host two simultaneous ATE events, one in each the English and Spanish communities.

This is also a good time to ask you to let us know what topics you'd like us to discuss in the future and we'll try our best to cover it.

Cheers!