storage-man
1 Copper

Turn on disabled CX600 port

Jump to solution

We use an EMC CX-600 and a CX0700 SP with four storage servers(RHEL4) in our SAN to teach the course "Storage Area Networks" at Michigan Technological University in Houghton, Michigan. This system has been in running for the past 4 years. It is a non production system used only for educational use. We use it only during the Fall Semester(Sept. -> December).

This is a dual fabric system using one 24 port EMC FC switch for each fabric. One CX-600 channel A port is connected to the top fabric, switch, one CX-600 channel A port is connected to the bottom fabric switch, same for the B channel ports(one to each fabric switch).

Last November the port B3 connection to the bottom switch failed as we were using it. The other fabric is still working, so we were able to complete the semester using that connection.

Now I don't know why the B3 port is off. The status is:

[sat3200student@cnsa-stor1 ~]$ naviseccli -h 172.20.189.200 port -list -sp

Information about each SPPORT:

SP Name:             SP A

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:61:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:60:EE:F0:20:19:08:00:88:60:EE:F0

SP Source ID:        6363411

SP Name:             SP A

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:60:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:0A:08:00:88:02:09:7A

SP Source ID:        6359571

SP Name:             SP A

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:63:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

SP Name:             SP A

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:62:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

SP Name:             SP B

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:69:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:10:08:00:88:02:09:7A

SP Source ID:        6361107

SP Name:             SP B

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:68:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

SP Name:             SP B

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6B:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

SP Name:             SP B

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6A:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

[sat3200student@cnsa-stor1 ~]$

So there are two channel A ports connected, this is correct(the other two Channel A ports are not connected). But there is only one Channel B port connected, that is port 1. Port B3 is connected to the switch but the status is link status down, disabled. But the cable is connected to the switch and I have tried replacing the cable with a new one(I did that twice, no change). So how do I turn the port B3 on?

The green link light is lit for B3(and for B1) which tells me that the switch is talking to the CX-600 port but the CX-600 disabled the port. How can I turn it on?

Danny Miller(danny@mtu.edu)

Labels (1)
Reply
1 Solution

Accepted Solutions
Highlighted
kelleg
5 Tungsten

Re: Turn on disabled CX600 port

Jump to solution

Please mark this thread as "Answered" - that will help others looking for similar issues. Glad you got it fixed. Dust is a demon.

glen

View solution in original post

Reply
9 Replies
Highlighted
umichklewis
4 Tellurium

Re: Turn on disabled CX600 port

Jump to solution

Hi Danny!

When I worked for U of M College of Engineering, I think a few of us came up and visited.  Hope all is well in Houghton!

Can you run "naviseccli -h 172.20.189.200 port -list -sfpstate" and "naviseccli -h 172.20.189.200 port -list -sp -all"?  This adds a bit of extra output, which might help determine if you have a bad SFP.

Let us know if that helps!

Karl

Reply
Highlighted
storage-man
1 Copper

Re: Turn on disabled CX600 port

Jump to solution

Yes, all is mostly well in Houghton!

Here is the output of both of the commands that you suggested. My system does not seem to support the "-sfpstate" parameter:

[sat3200student@cnsa-stor1 ~]$ naviseccli -h 172.20.189.200 port -list -sp -all

Total number of initiators: 11

SP Name:             SP A

SP Port ID:          1

Registered Initiators:        3

Logged-In Initiators:         1

Not Logged-In Initiators:     2

SP Name:             SP A

SP Port ID:          0

Registered Initiators:        2

Logged-In Initiators:         1

Not Logged-In Initiators:     1

SP Name:             SP A

SP Port ID:          3

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

SP Name:             SP A

SP Port ID:          2

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

SP Name:             SP B

SP Port ID:          1

Registered Initiators:        3

Logged-In Initiators:         1

Not Logged-In Initiators:     2

SP Name:             SP B

SP Port ID:          0

Registered Initiators:        1

Logged-In Initiators:         0

Not Logged-In Initiators:     1

SP Name:             SP B

SP Port ID:          3

Registered Initiators:        2

Logged-In Initiators:         0

Not Logged-In Initiators:     2

SP Name:             SP B

SP Port ID:          2

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

Information about each SPPORT:

SP Name:             SP A

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:61:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:60:EE:F0:20:19:08:00:88:60:EE:F0

SP Source ID:        6363411

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:60:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:0A:08:00:88:02:09:7A

SP Source ID:        6359571

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:63:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:62:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:69:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:10:08:00:88:02:09:7A

SP Source ID:        6361107

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:68:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6B:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6A:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

[sat3200student@cnsa-stor1 ~]$ naviseccli -h 172.20.189.200 port -list -sfpstate

usage: port -list <-hba|-arrayhba|-sp|-gname groupname|-uid UID> <-all>

                  <-arraycommpath> <-failovermode> <-unitserialnumber>

                  <-mac> <-initiatorcount>

            -removeHBA <-o> -hbauid HBAUID|-host host|-hostip ip|-all

            -diagnose <-o> -sancopy <-clean>|-host

Invalid command line parameters: -sfpstate

[sat3200student@cnsa-stor1 ~]$

I did think that perhaps an sfp has failed, but if that is the case then I have 3 of them failed on the b channel because I have all four of those ports plugged into a switch and they all show disabled states even though all of them are lighting the green connection light on the CX-600 rear panel. I grabbed a pair of long nosed pliers and tried to pull(gently!) one of them out but no luck. Are those field replaceable? I am only using two of them on the A channel so I could swap one of them to B3 if that is possible.

Danny

Reply
Highlighted
kelleg
5 Tungsten

Re: Turn on disabled CX600 port

Jump to solution

On the CX600/700 the SFP on the port can not be changed, requires a SP replacement.

You know that SPB port 1 is good, so try that cable in port 3 (the cable from switch port for SPB1 to SPB3). If it still doesn't come up, try rebooting SPB. I think I remember something about the ports locking on the CX600 and needing to reboot SP.

glen

Reply
Highlighted
storage-man
1 Copper

Re: Turn on disabled CX600 port

Jump to solution

I switched the cables that were connected to ports B1 and B3. When I did that, the switch port connected to B1 came up but the one connected to B3 did not. Port B3 was still disabled. I rebooted channel b, port B3 still disabled. I powered down the entire array(this is an educational machine and we are not yet using it for lab experiments) and brought it back up, no change, B3 still disabled, B1 enabled.

So I guess that the port B3 sfp is shot. Ports B0 and B2 are also disabled, and won't light a switch port connected to them. So I guess that B0, B2 and B3 must all have defective sfp's. B0, B2 and B3 all show the green link light on the CX600 when I plug in a cable, but none of them will come out of the disabled state. It seems strange that all three of them act the same, but this is an older unit so perhaps all three have failed transmitter LED's in the SFP.

I do have a "spare" CX600 so the next step is to do some testing on my "spare" channel b unit to see if it has usable ports.

Thanks for all of the information, very useful.

Danny Miller

Reply
Highlighted
kelleg
5 Tungsten

Re: Turn on disabled CX600 port

Jump to solution

Did port B3 work in the past? Is there a zone on the switch for B3? I sort of remember that for the port to come on-line, you need both a cable plugged in the SP port that is connected to a switch port and a zone for that port on the switch.

glen

Reply
Highlighted
storage-man
1 Copper

Re: Turn on disabled CX600 port

Jump to solution

This system worked for over 4 years in its present configuration. I have dual fabrics(each fabric is one switch) for redundancy, the fabrics are not connected together. The port B3 went to the disabled state sometime in November 2014. Since we have dual fabrics, all of the storage servers still have connectivity via the other fabric so I finished out the fall semester using that.

The switches are zoned using the "single initiator zoning" plan. Each storage server HBA is zoned to one port on the CX600 and one port on the CX700(we have two storage processors in our SAN). I checked all of the zoning, and especially the zone that contains B3. It looks fine to me. I didn't do anything with the zoning last fall around the time that port B3 went disabled.

Their are four zones in our zone set for this fabric, one for each storage server. Each zone contains the WWN of that storage server HBA, one CX600 port, and one CX700 port. For the zone that contains B3, I just reactivated the zone, in case something had somehow changed. It didn't help.

As a test, I just connected port A2 to a switch port using a new cable. I had not been using A2, but when I connected it to the same switch the switch port immediately came up, and when i checked the port status for A2 is is now "up" and enabled. It was 'down" and "disabled" before the connection. It did that with no rebooting of the SP and no zoning of the switch, so I don't think that the zoning is needed to enable a port.

Hmm, it still seems strange that I have ports B0, B2 and B3 all connected to switch ports but none of them will bring up the switch ports and all sit there "disabled". The link lights on the CX600 for B0, B2 and B3 are all green but the switch lights on the other ends of those cables are all dark. How can all three ports have dead hba's?

Reply
Highlighted
kelleg
5 Tungsten

Re: Turn on disabled CX600 port

Jump to solution

It's odd that three of the four ports failed, but that's what it sounds like. You mentioned that you had a spare CX600 - the SP's are identical for SPA and SPB, so you could use either one to replace the SPB SP. Replacing the SP on the CX600 is pretty easy. First, disable System Write cache before you remove the SP - this will dump all the Write cache to the disks. Then you can pull SPB out, it's hot swapable. Disconnect the cables and reconnect to the new SP and then re-insert, it should start booting at the point.

If you want to feel safer, then power down the array first - use the power switch on the SPS backup power supplies, then should take about 2-3 minutes to power down - disable Write cache first just to be safe.

glen

Reply
Highlighted
storage-man
1 Copper

Re: Turn on disabled CX600 port

Jump to solution

I did some testing on my "spare" cx-600 yesterday. The A channel unit is not working, but the B channel unit works with all four front end ports usable(they all light up ports when connected to a switch).

So this morning I powered down my CX-600 and replaced the B channel SP with my B channel spare unit. It works, all four front end ports(B0 through B3) are now functional. The old unit was very dusty, dust was heavy on the heat sinks of the processors and on the heat sink of the large IC's mounted on the motherboard. I remember removing a lot of dust from the back and front of the SP enclosure when I received this unit 6 or 7 years ago, but I didn't think to open up the SP and remove the dust that was certainly limiting airflow to the internal heat sinks. I intend to power this unit back down, remove the A SP and clean out the dust.

Thanks much to all of you for your help. Glen, your detailed instructions were very useful.

Danny Miller

Reply
Highlighted
kelleg
5 Tungsten

Re: Turn on disabled CX600 port

Jump to solution

Please mark this thread as "Answered" - that will help others looking for similar issues. Glad you got it fixed. Dust is a demon.

glen

View solution in original post

Reply