Highlighted
markgreen1966
1 Copper

Best Practice SAN connectivity

Hi, I have been in a debait with another engineer on how we should connect our SAN fabric to our servers and CX4 series SAN array, via dual Cisco Fabric switches. i am looking for some input , to ether validate my argument or see if he is correct :-

we have currently :-

  • All SP-A connections going to First switch which is Fabric A
  • All SP-B connections going to second switch which is Fabric B
  • Servers are Dual HBA cards. one connection from Fabric A and one from Fabric B

So if one switch Fails, the servers loose connectivity to the SP-B controller, and all the disks trespass.

he keeps mentioning that EMC says your not supposed to mix SPA and SPB connections in one fabric.

and that when disks trepass thats a normall function and quiet exceptable for reduncancy.

Ok yes this does work.

My understanding is that you can mix Both SP-A and SP-B connection on one Fabric.

So each HBA connection on the server can see paths to both SP-A and SP-B

  • So SP-A1 connects to First Switch Fabric A
  • SP-A2 connects to Second Switch Fabric B
  • SP-B1 connects to First Swithc Fabric A
  • SP-B2 connects to Second Switch Fabric B

and so on.

Hence if a switch fails, the server can still see connections to both A and B controllers.

Plus, if some servers only have a single HBA they can still see both controllers.

Thoughts please !!

Labels (1)
0 Kudos
11 Replies
dynamox
6 Gallium

Re: Best Practice SAN connectivity

your thought process is absolutly right, you want to have SPA and SPB ports on each fabric so that each HBA is zoned to both, IE:

Fabric A Zones:

HBA1 - SPA0

HBA1 - SPB3

Fabric B Zones:

HBA2 - SPB0

HBA2 - SPA3

not only it does it not need to trespass LUNs in case of fabric failure, you get better load-balancing as each HBA will be used for traffic to both fabrics.

RRR
5 Tungsten

Re: Best Practice SAN connectivity

Listen to Dynamox, he is right! Connect SPA and SPB to both fabrics. Also consider that when you have mirrorview using SPA0 and SPB0 for replication that you don’t want these 2 ports on the same fabric, so when somehow 1 fabrics fails the other one still offers mirrorview connectivity.

So SPA0 / SPB1 on one fabric and SPA1 / SPB0 on the other is good.

0 Kudos
Sridhar246
2 Iron

Re: Best Practice SAN connectivity

You are correct sir, zone ports on SPA and SPB to both the switches. Tresspass is acceptable (even though I don't like tresspassing luns during production hours) but lets say for some reason (with the config you have) one of the switches and all your luns present on SPA tresspass to SPB. Now lets say before SPA and SPB were to be running at close to 50% utilization after tresspass everything is on SPB with close to 100% SP utilization adversly affecting performance.

0 Kudos
dynamox
6 Gallium

Re: Best Practice SAN connectivity

CPU utilization has no relevance on how port connectivity is established.

0 Kudos
Sridhar246
2 Iron

Re: Best Practice SAN connectivity

Thing I wanted to say is with the config he has in case of one switch failure, all the luns owned by SPA (for example) will be tresspassed to SPB which will shoot up the SP utilization of SPB which might affect performance. Please correct me if I am wrong.

0 Kudos
dynamox
6 Gallium

Re: Best Practice SAN connectivity

looking at the current config i agree, now with ALUA that could be masked because you could still talk to SPA through the cmi bus.

0 Kudos
AndyHengst
1 Copper

Re: Best Practice SAN connectivity

An obvious question would be "Where does it say that?" (keeping SPA on fabric A etc) If that's documented I'd like to know!

We also zone each SP to both fabrics. To put it another way, I'd rather have powerpath look after the LUN visibility for me than have a trespass do that.  There might be another reason: if a switch on fabric A fails, a LUN on SPA *may not* trespass. There is no problem with SPA and I"m not trespassing the LUN, so why would it trespass to SPB?  This is something you should test to be sure. I believe you'd simply lose connectivity.

A bit off topic, but related to zoning two fabrics: we made our life a bit easier (and less error prone) by using the same zone set for both fabrics. Even though half the zones are not needed in one fabric, and the other half are not needed in the other fabric, you maintain only one zone set and you CANNOT accidently activate the wrong zone set to a fabric.

0 Kudos
dynamox
6 Gallium

Re: Best Practice SAN connectivity

what do you mean you have one zoneset for both fabrics, can you elaborate ?

Thanks

0 Kudos
AndyHengst
1 Copper

Re: Best Practice SAN connectivity

Hi dynamox,

we use SMC with one Zone Library and one Zone Set with sufficient zones to allow all required connections to work. We activate this zone set to Fabric A, and a few minutes later activate the same set to Fabric B.  Each fabric will have exactly half unknown/unused WWN zones, but that's OK.

Here are some typical zone names:

blackbox_A_CX1234_SPA0

blackbox_A_CX1234_SPB1

blackbox_B_CX1234_SPA1

blackbox_B_CX1234_SPB0

... all of them go to all switches.

(editSmiley Happy But if your environment is nearing the maximum capacity for zones then it might be time to separate these out.

0 Kudos