Shutdown and move Cisco 9506 switches

Brad,
I have done this a few times unfortunately with MDS switches. I say unfortunately because it is never fun to relocate a data center. Last year we moved two production local data centers into one large production data center. The switches were the easiest part of the move

Make sure you backup each switch:

switch# copy run start
switch# copy start ftp:switch1_config.txt

We moved our main production data center in one weekend (300 servers, 60 SAN-attached servers, two MDS9509, four CLARiiONs, one Centera) so were not moving one switch at a time. Using that process we shut down all hosts first, then all arrays, and after copying off the startup config for the switches we powered them down. So there was no host i/o to deal with when shutting down the switches. Our secondary production data center was much smaller, two MDS9216s, one CLARiiON and three hosts, so it was very easy.

Since you are going to be shutting down a switch that is in use you will want to be more careful. The shutdown command is for ports, not the entire switch. You could use the reload command without any arguments which would restart the switch. You could then power off the switch manually as soon as it shuts down and before it boots up. If you are doing this during a low i/o period (I hope) then you could just power off the switch after copying off the config. These are cisco switches and can be powered off manually without any other command run first.

If you are using pWWN zoning and not port zoning you don't have to worry about getting the cables exactly in the same spot. I even used the move to organize the cabling better and changed a lot of the cabling locations. Purchasing new cables for all hosts and storage and running it all before hand will save you tons of time. This made things much easier for us as you can imagine.

Hope this helps you. Did I answer all your questions?

Aran

Brad,
Yes, you can just toggle the black switches when you are ready to powerdown. Do you have the luxury of performing a test of the process? You could schedule a time to just shutdown one of the switches. See how all your hosts deal with the outage (find out if there is any multipathing that is not setup right ) and then power on the switch again.

The MDS switches are real solid in my experience and I have had to move 9509s a few times and have yet to have any issues.

Aran

What about port security ? Although not commonly used, it is possible to only allow 1 wwpn to connect to a specific port. I would label the cables anyway, just to make sure.

I am more familar with AIX than Windows although we run both. When we move our MDS switches I will need to know which HBA goes to which switch. This becomes more difficult because I have never visited this remote data center in Texas. For the AIX host below I know it uses HBAs fcs3 and fcs6 to connect to the DMX. Below I have run the lsslot command and can see fcs3 is in slot P1-I5 and fcs6 is in P2-I5. When the time comes I can instruct the onsite operator to unplug the old cable from the correct slot and plug in the new cable that runs to the new location of the mds1 switch.

root@wfscr05:/#lsslot -c slot
# Slot Description Device(s)
U1.9-P1-I1 Logical I/O Slot pci7 fcs0
U1.9-P1-I2 Logical I/O Slot pci8 ent0
U1.9-P1-I3 Logical I/O Slot pci9 fcs1
U1.9-P1-I5 Logical I/O Slot pci16 fcs3
U1.9-P1-I10 Logical I/O Slot pci11 sisscsia0
U1.9-P1/Z1 Logical I/O Slot pci10 scsi0
U1.9-P1/Z2 Logical I/O Slot pci22 scsi4
U1.9-P2-I1 Logical I/O Slot pci12 fcs2
U1.9-P2-I2 Logical I/O Slot pci13 ent1
U1.9-P2-I5 Logical I/O Slot pci25 fcs6
U1.9-P2-I8 Logical I/O Slot pci15 ent2
U1.9-P2/Z1 Logical I/O Slot pci14 scsi1
U1.9-P2/Z2 Logical I/O Slot pci29 scsi5
root@wfscr05:/#

Windows is more difficult for me. There are 2 HBAs in each server. At the switches each cable is labeled hba1 or hba2. Is there a simple way in windows to find out which slot hba1 is in? I have not been able to find anything like the AIX lsslot command.

I usually just open up the HBA manager software on the host, list the pwwn of the hbas and compare that to the zones for the host on each switch. That will tell you which HBA is plugged into which switch, and then you can flash the LEDs on the HBA to determine which physical slot is associated with the HBA.

And the port security (hard zoning) in my opinion is not worth the effort. If your data center is unsecured to the point that anyone can plug into a switch you have bigger problems

port security is NOT hard (port) zoning !

Port zoning means you put 2 ports in a single zone and let the wwpn communicate with each other, no matter what you plug in there.
Port security means that the wwn of a device can be allowed as the only wwn able to use that port. After having set the security, you can still use wwn zoning !!

check out this document on the Cisco website: http://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/rel_1_x/1_3/fm/configuration/guide/SecuPort.html

Sorry Rob I have heard people use the term port security intermingled with hard zoning before and didn't realize you were actually referring to Cisco port security.

But I would not implement port security and restrict a pwwn to a single port, but rather a range of ports. That way you can move the cable if needed (or by mistake) without the administrative headache of reconfiguring your port security and the fabric is secured to trusted pwwn's.

One more question. Currently mds1 has vsan100 and mds2 has vsan200. All hosts and storage arrays are redundant on both switches. For example a host will have 2 hbas with one going to mds1 and may zone to cx380_spa1. The other hba may go to mds2 and zone to cx380_spb1. Both switches uses two ports to form an ISL which is vsan300.

We will move mds1 one night and then move mds2 a different night. The night that we move mds1 we will remove all old cables including the two ISL ports. Mds2 will continue to run and all hosts and storage will remain up on mds2 while we power off and move mds1.

We have never removed thse ISL cables before. I am not sure what will happen on mds2 when we remove the ISL cables on mds1. It may take a while to get mds1 racked up in the new location. Once it is back up we will immediately plug in two new cables for the ISL. Should we anticipate errors/problems with mds2 when we remove the ISL cables?

Brad,
I will let someone else answer that question, I don' have much experience with ISLs.

Question for you though, your two MDS906s are linked via ISL? So you don't have redundant/separate fabrics? Is there a reason you have the ISL between your switches?

It might happen that a port will go "error disabled". This will take a little while before happening (I'm not sure if it's 1 hour or 2 or maybe even half an hour), but when a port gets in this state, you'll need to "shut" and "no shut" the ports on both sides to get the ISL's up and running again. The portchannel wil restore itself after that.

Brad,
I am not sure either why they would have implemented two switches with an ISL without a specific solution that required it. Ideally you would have each fabric logically and physically separate to prevent any fabric wide problems from affecting your entire infrastructure. You have physical redundancy with your current setup, but with the ISL there are shared logical components.

If you have the FM license you can manage both fabrics at the same time. Other wise you have to open one at a time. Not sure how the ISL affects this, but IMHO it is not a good reason to connect your fabrics.

AranH - Good question about the redundant fabrics and ISL. The two fabrics were implemented by EMC with an ISL. The first switch is vsan100 and the second is vsan200 and the isl (2 ports on each switch) is vsan 300.

I'm not sure why EMC implemented it this way. Every host and every storage port has it's redundant counter part in the other switch/vsan. I'm not sure why we would need an ISL for this. Maybe it has something to do with Fabric Manager being able to see both fabrics at the same time.

Any thoughts would be appreciated.

I would assume the ISL was created for the ability to grown as the SAN matures. You can always just block the ISL connection between the switches. What usually happens if you isolate the switches though is that you have one resource on one switch you need on the other switch. You can always recable but having an active ISL you can use is faster.

Thank you.