Highlighted
seanka1
1 Copper

PowerPath Connection to CX4

Jump to solution

Hello All,

Mods, please direct this Discussion to the correct venue if I have placed it in the wrong location.

I have a host that has started behaving very oddly.  Here is my setup:

  • Windows Server 2008 R2 Host
  • PowerPath 5.5 SP1 (Unlicensed)
  • Server acts as a Media agent for CommVault
  • Server has approximately 15 LUN's mapped to it
    • Two LUNS are NTFS
    • 13 LUNS are VMFS
  • Using CommVault VSA to backup the VMWare VM's via SAN Mode
  • QLogic Two port FC
  • HBA, each port is individually zoned to each SP

  • The server also has 6 iSCSI disks that are mapped to NON-EMC storage

The host has been exhibiting the following behavior:

  1. When my CommVault Backups kick off, I have been getting a tremendously high number of EMC Alerts indicating that "The storage system received too many events in this poll cycle to display here.  This does not necessarily indicate a problem."
  2. When I restart the host, if both SPA and SPB are connected to the host, the server goes into a perpetual reboot cycle, and never will boot into windows.  If I unplug one of the FC cables, the server boots successfully into the OS.
  3. In PowerPath, if I expand Path Management > Storage Arrays > Storage ID; SP A1 has all of the Disks (NTFS and VMFS) - Mode: Active, however SP B1 ONLY has the two NTFS Volumes - Mode: Unlicensed.

My questions are:

  1. Other than Windows updates, nothing has changed on this host from a configuration perspective .  I cannot figure out why it all of a sudden started this behavior.
  2. Why would SP B1 only two disks whereas SP A1 shows all disks?  Is this part of the problem?
  3. I understand that PowerPath is unlicensed - does this mean that the server would not be able to have disks that are on both SPA and SPB on the CX4?  Since PowerPath is unlicensed, must all disks reside on a single SP for the host to be able to properly access the data on those disks?
  4. When viewing the list of disks in PowerPath, it says that the current owner of ALL disks is SPA.  However, when in Navisphere, it indicates that the majority of the VMFS LUN's are on SPB whereas only two NTFS and two VMFS LUN's are on SPA, can this be a contributing factor?
  5. Has anyone experienced the perpetual reboot scenario when you have both paths connected to a Windows Host?

Thanks in advance for your advice - I will be glad to amend/append as needed.

Message was edited by: seanka - Formatting issues

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
dynamox
6 Thallium

Re: PowerPath Connection to CX4

Jump to solution

if you are presenting VMFS LUNs to the Windows server storage group, i would expect to see them listed just like your two NTFS LUNs (unless CommVault is doing some kind of magic and hides one path).

The fact that the VMFS LUNS are not presented to the second HBA is what I am wondering is causing the issues.  Plus, according to Navisphere there are about 10 luns that are sitting on SPB.  So, is my guess correct that when this server tries to backup the data on those luns located on SPB, it is tresspassing the luns to SPA so that it can complete?

if this in fact is happening ( it could), you can get into trouble (trespass storm), where your backup server tresspasses the LUN to SPA and then VMware trespasses it back to SPB ..so you get into this tug of war. I would make sure each HBA is zoned to SPA and SPB.

0 Kudos
10 Replies
dynamox
6 Thallium

Re: PowerPath Connection to CX4

Jump to solution

can you get a trial license for PowerPath ?  Each HBA is zoned to SPA and SPB ?  What if you completely block access for one HBA ?

0 Kudos
RobertDudley
3 Argentium

Re: PowerPath Connection to CX4

Jump to solution

Unlicensed powerpath only provides failover at the array level not the host hba level.  All I/O will use one path to the storage system.  If luns are owned by both SP's when this host tried to access them for backup it may be causing the lun's to trespass to the other SP.

As dynamox suggested license powerpath or remove it and use the built-in DMP.  I would suggest getting a PP license.

0 Kudos
seanka1
1 Copper

Re: PowerPath Connection to CX4

Jump to solution

Thanks Guys,

So, to clarify.  In theory I need to have a licensed copy of PowerPath on this host since I need to access the LUNS on both SP's.  Since this single host is accessing LUNS that are shared amongst other hosts, it will need access to both SP's concurrently. 

If I have PowerPath licensed, and the host tries to access a disk on SPB OR SPA at any given time, it will be able to communicate with both of them?

0 Kudos
dynamox
6 Thallium

Re: PowerPath Connection to CX4

Jump to solution

here is an example for you (this is Linux but same concept)

Pseudo name=emcpowera

CLARiiON ID=APM00000000000 [server07]

Logical device ID=600601609E402D00008BDBBA4612E211 [LUN 366]

state=alive; policy=BasicFailover; priority=0; queued-IOs=0;

Owner: default=SP A, current=SP A    Array failover mode: 4

==============================================================================

--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---

###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors

==============================================================================

   1 fnic                     sdb       SP B3     active  alive       0      0

   1 fnic                     sdc       SP A0     active  alive       0      0

   2 fnic                     sdd       SP A3     unlic   alive       0      0

   2 fnic                     sde       SP B0     unlic   alive       0      0

PowerPath is not licensed but each HBA is zoned to SPA and SPB so you can access LUN on either SP using HBA 1 but from failover and load-balancing perspective the second (unlic) HBA is not in use. So if you were to experience port failure on the switch, HBA 1 failure, cable failure ..this host would lose total storage access.

Can you post output from "powermt display dev=all"  ? (scramble your serial numbers/host names)

seanka1
1 Copper

Re: PowerPath Connection to CX4

Jump to solution

Dynamox,

Please see attached - I hope it can be of additional help. 

0 Kudos
dynamox
6 Thallium

Re: PowerPath Connection to CX4

Jump to solution

interesting, VMFS LUNs are not presented to the second HBA (even not licensed). Is this VCB type of backup, where you present VMFS LUNs to a windows box and then spin vmdks to tape.

You have a couple of options:

1) Best case - get PowerPath license, get better load-balancing, throughput, failover. Zone each HBA to SPA and SPB.

2) If not licensed - might as well disconnect the second HBA, it's not going to buy you anything. Zone first HBA to SPA and SPB and see if you problem re-appears. Right now , based on the output your provided, if LUN trespasses to SPB, you are hosed because HBA 2 (unlicesed) is the only path to that SP.

seanka1
1 Copper

Re: PowerPath Connection to CX4

Jump to solution

HI Dynamox,

That is correct.  The VMFS luns are presented to the windows box. Then, the backup software (CommVault) takes a snapshot of the virtual machine and backs up the VM.  The backups are done via FC directly connected to the VMFS Datastore.

The fact that the VMFS LUNS are not presented to the second HBA is what I am wondering is causing the issues.  Plus, according to Navisphere there are about 10 luns that are sitting on SPB.  So, is my guess correct that when this server tries to backup the data on those luns located on SPB, it is tresspassing the luns to SPA so that it can complete?

Is there a reason that you can think of that PowerPath would see the the VMFS luns on the second HBA?

0 Kudos
dynamox
6 Thallium

Re: PowerPath Connection to CX4

Jump to solution

if you are presenting VMFS LUNs to the Windows server storage group, i would expect to see them listed just like your two NTFS LUNs (unless CommVault is doing some kind of magic and hides one path).

The fact that the VMFS LUNS are not presented to the second HBA is what I am wondering is causing the issues.  Plus, according to Navisphere there are about 10 luns that are sitting on SPB.  So, is my guess correct that when this server tries to backup the data on those luns located on SPB, it is tresspassing the luns to SPA so that it can complete?

if this in fact is happening ( it could), you can get into trouble (trespass storm), where your backup server tresspasses the LUN to SPA and then VMware trespasses it back to SPB ..so you get into this tug of war. I would make sure each HBA is zoned to SPA and SPB.

0 Kudos
seanka1
1 Copper

Re: PowerPath Connection to CX4

Jump to solution

The VMFS Luns show up in Disk Management on the Windows Server as "disks" but don't show up My Computer as expected.

The Tug of war that you are describing sounds exactly like what is happening.  I think I am going to get a trial license of PowerPath and test it.  If it resolves the problem, I think I'll have a good justification for the License.

Dynamox - many thanks for your advice on this.  You provided insite and an explanation that unfortunately EMC Support couldn't.

Sean

0 Kudos