Removing LUN from Server - PowerPath for Linux - powermt issue

unfortunately i can't provide a solution but to share your frustration. We always have these issue, even on the latest 5.x RHEL boxes. The other day we had to upgrade PowerPath from 5.1 to 5.5 and PowerPath kept complaining that devices were in use, how can they be in use if all file system were un-mounted and volume groups were exported. Stopping PowerPath service would not help either. I literally had to disable ports on the switch, bounce the box and only then we were able to upgrade PowerPath. Linux after so many years is still so weak when it comes to LVM and disk subsystem. You very rarely see these issues on AIX/HPUX/Solaris ..real *nix boxes

I had same issue formerly. This happens when you removed devices on Clariion without cleaning on host.

What I did is just give back the lun (id must be same) and remove using powerpath commands, then clean up on storage array.

This might help without rebooting ...

Hello.  Have you contacted anyone from EMC's Customer Support about this issue?

Thanks, Brion

Please keep me informed on your progress. I’d like to see this issue researched.

Thank you for the responses.  I have not opened a case with EMC, mostly because I suspect, based on some past experience, the answer will be reboot.

Today I made the same change on three more servers (another RAC cluster). Befor unpresenting the LUNs from the Clariion I issued the 'powermt remove' command, but the answer was -

#powermt remove force dev=emcpoweri
Cannot remove device that is in use: emcpoweri

Even though there was nothing mounted on the device.

Interestingly, the issue on these three nodes is consistent with what happened on the first three. One path remains on the EMC pseudo device in a display -

#  powermt display dev=emcpoweri
Pseudo name=emcpoweri
CLARiiON ID=APM00083101547 [bkvdwracp RAC cluster]
Logical device ID=6006016072C020003E7F9D2EC585DE11 [bkvdwracp - /exports on bkvdw03p - LUN407]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 lpfc                      sdh       SP A4     active  dead       0      1

I cannot reboot at this time, but it does not appear to be causing any issues in this state.

I have two more clusters to do this to. They are each only two nodes. Maybe I will find out more before then!

Thank you for all the helpful posts. i have performed the same task on eight servers in three RAC clusters, all with the same result. I have tried unpresenting the LUN's before cleaning up PowerPath and tried the 'powermt remove' before unpresenting..with no difference. PowerPath thinks one of the paths is still busy even though the filesystem is unmounted. It will removed all the dead paths for the device except for one in every case -

Pseudo name=emcpowerd
CLARiiON ID=APM00083101547 [bkvoltpda RAC cluster]
Logical device ID=6006016072C020007EBABDD310E0DE11 [bkvoltpda - /exports on bkvoltp01da - LUN504]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   1 lpfc                      sdaf      SP B5     active  dead       0      1

Atleast it is consistent! I expect these will finally clean up on a reboot. I have one more cluster (two servers) to do.

Thanks,

Michele

On the final time doing this the result was consistent and one dead path remains.

I am planning to remove LUNs from a standalone server next week and am curious to see if the result is still the same.

I will update this forum with my results.

Thanks.

since you mentioned RAC cluster, are those LUNs part of ASM disk group? if yes, the devices cant be released till you bunce the ASM on the node.

Thanks for your response. No these are not ASM disks, strickly LVM.

Although I am curious to see if perhaps the clusterware may be the culprit. Tomorrow I will be removing a LUN from a standalone server...although same OS and version of PowerPath.

I will update with result.

when you said you tried to remove when the device is mounted, but was the device part of the VG that time? were u able to do a vgexport ?

Today I removed a LUN from a standalone Linux Server. I followed the same steps that I did on the previous RAC nodes -

umount the directory

lvremove, vgchange, vgremove

powermt remove dev=emcpowerc <== this time there were no error messages of device in use

powermt release

Un-presented LUN from SAN

powermt check  <==saw all four dead paths and removed them'

powermt save

vi /etc/fstab to remove old entry

This time it worked perfectly! So, my suspicion is that the RAC clusterware is somehow holding the last path 'in use' and not allowing the dead path to be removed. Until, I assume, I reboot.

Pseudo name=emcpowere
CLARiiON ID=APM00083101547 [bkvoltpp RAC cluster]
Logical device ID=6006016072C02000E41689897DDFDE11 [bkvoltpp - /exports on bkvoltp01p - LUN454]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   1 lpfc                      sdaf      SP B5     active  dead       0      1

[root@bkvoltp01p ~]# powermt remove force dev=emcpowere
Cannot remove device that is in use: emcpowere

Thank you for all the responses.

Michele, firstly thank you for sharing your experiences throughout.  Not sure if it would provide anything meaningful and help identify what is holding onto it, but does lsof on the device provide anything useful to see what process might be holding onto it?  Run on both the dev files (using the output above):

lsof /dev/emcpowere

lsof /dev/sdaf

Hi Chris,

I tried the lsof and nothing was returned. 

These LUNs were presented from the SAN to all three nodes in the RAC, but each was only configured with lvm and mounted on one node only. I am not sure if that would make a difference...

Other than waiting to schedule a reboot of all the servers, I am not sure what else to do. It is not causing any issues I can see.

Thanks!

since you have mentioned RAC, is the shared VG(LVM) are you following here?