I have a Windows 2003 host that is giving me some strange behavior in PowerPath. This host has 2 HBAs - Emulex LP982s. I noticed that one of the adapters no longer shows up in Powerpath. Yesterday I saw 2 adapters, but the second was showing no connections, and I also check in the OCManager application, and it reported the adapter as down. After trying a reboot on the host, the second adapter disappeared completely from PowerPath.
I've seen this behavior in the past on other hosts, except usually I see this accompanied by the host missing initiators in Connectivity Status on the array (Clariion CX3-80), and the ~management storage group being present. Previously I've fixed this by either re-initializing the initiators for the host, and on a few occassions I ended up replaceing the suspect HBA. This time however, in Connectivity Status I'm seeing 8 paths for the host logged in and registered as expected.
Each HBA is a single port and connected to each SP on the Clariion.
Since the array is seeing all of the paths, this would seem to me like a PowerPath issue on the host machine, but I haven't been able to isolate this. I restarted both management servers on the Clariion SPs and still see all the paths, so this "seems" to indicate that the array has a good connection across all 8 paths. What really has me confused with this situation is that both Powerpath and OCManager are agreeing that one of the HBAs is down. If the HBA had failed, I would expect to see half of the paths missing on the array.
I've performed several resets on the HBA, and this morning I'm due to try flashing the firmware again so I can cross that off my list later. In Engineering Mode on the CX3-80 I checked the connectivity status for the host and performed a Reconnect, and all of the paths there still show to be green.
If anyone has some other ideas to try for troubleshooting, or maybe has encountered this behavior previously, I would appreciate any assistance.
It seems you already covered all possible issues. I can't think of any more things to check or do. Could it be a driver issue (ofcourse not), a SAN issue (all paths active in connectivity status, so ofcourse not)... A faulty FC patch cable ?
I'd say it's the HBA again. I've had HBA problems again and again on 1 of my servers in the past, so it certainly could be 1 HBA multiple times. Maybe it's a PCI slot issue ?
I suspected the patch cables involved too. The overall physical connection has each HBA with a FC patch to a patch panel, then the patch panel to a pair of Cisco MDS switches. Each HBA is connected to the same port on each switch, so yesterday I tried reversing the cable connections on the switchs, and that resulted in losing all of the drive paths on the host which really confused me.
I also tried a long single FC cable to run from the HBA to the switch port and the connection still registered as down. I tried the same cable on the working HBA and that connection stayed up so I know the long cable is good.
Am I wrong in expecting the CX3-80 to only see 4 paths logged in and registered if the HBA had actually faulted? I feel like if I saw only 4 paths, I could confidently say that the HBA is bad after having tried to troubleshoot the other angles. I am hoping to replace the HBA later today and will see if that changes the behavior, but really I'd like to try and understand what would cause this path mismatch between what the host sees and what the array sees.
One more note - I've had problems with GBICs failing on the switches before, so we replaced those too in the initial troubleshooting steps. I tested sevral GBICs against known good ports and they worked fine, so the issue in this case wasn't the GBIC. As far as issues in the connection path go, I could also throw suspicion at the switch port itself, but again that doesn't add up to me since the array sees 8 registered paths back to the host.
Thanks for the reply,
Try updating PowerPath to 5.5 - there are a number of fixes in the new versions, some affect connectivity.
Also, check on Emulex web page, under the EMC section for the HBA drivers for your model and make sure you have any listed Microsoft hot fixes installed as well.