Removal of devices in powerpath 4.5.3

Hey, we have had this same problem more than once. Now I see that there is some solid process using "remove" before unmaksing them but let's say that "reboot" is not an option... Oracle RAC. We unmasked 2 LUNs after dropping them in ASM but apparently they were not totally flushed, so now we have all paths like this for the remaining LUNs:

Pseudo name=emcpower**
CLARiiON ID=APM00062200019 [prdrac02]
Logical device ID=6006016066AC1A00EAC27B8FC166DC11 [LUN 93]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
### HW Path                 I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 UNKNOWN                   unknown   SP B3     active  alive      0      0
   0 UNKNOWN                   unknown   SP A2     active  alive      0      0
   0 UNKNOWN                   unknown   SP B2     active  alive      0      0
   1 UNKNOWN                   unknown   SP B1     active  alive      0      0
   1 UNKNOWN                   unknown   SP B0     active  alive      0      0
   1 UNKNOWN                   unknown   SP A1     active  alive      0      0
   0 UNKNOWN                   unknown   SP A3     active  alive      0      0
   1 UNKNOWN                   unknown   SP A0     active  alive      0      0

And, for the 2 that were unmasked, they can't be "removed" via powermt:

-bash-3.00$ sudo /sbin/powermt display dev=emcpowerac
Pseudo name=emcpowerac
CLARiiON ID=APM00062200019
Logical device ID=6006016066AC1A00226BC441BE2DDE11 [LUN 117]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP B
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
### HW Path                 I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 UNKNOWN                   unknown   SP B3     active  dead       0      1

-bash-3.00$ sudo /sbin/powermt remove dev=emcpowerac
Cannot remove device that is in use: emcpowerac

Is there any way to force a cleanup of all the /dev/sd devices and fix powerpath config without taking them offline (rebooting)?

Yeah, no, that never works. Neither does "remove force" or unmanage, or any other powermt command.

I saw that post. I asked the DBA and he said that if we "bounced" ASM that all disks would be brought offline. Maybe ASM works differently that in HP-UX, I don't know much about it. We have in the past been able to do this process without any hang-ups, but occasionally it fails to release them the way you would think it should.

thanks

Hi huniedi,

I don't know if this is related to your case but some history ASM/HP-UX:

HP MC/SG didn't support ASM in a cluster because once ASM activate a DiskGroup even after deactivation it doesn't release the device files.

Currently Oracle released a patch that fixes this issue and ASM is supported in MC/SG.

So the bottom line: you may need to stop ASM, remove the devices from PP and then bring up ASM (yes, you have to shut down all DBs that use ASM)

We can all agree that things never act exactly the same way on different OS's. So, the problem here is specific to Linux. Huniedi is seeing the exact same thing we've experienced (and are right now on a production system) many, many times. We never had this problem on any of our AIX boxes running PP, in fact we've never had it on any of our Solaris boxes or if we did it did not manifest quite the same way. So, our DBA did some digging in OTN and found this:

There is a documented issue about ASM holding onto disks after they are dropped.

Oracle says it is not a bug and explains why they feel this way.

Goal

Why are the ASM devices still opened by Oracle foreground and background processes even after they have been dropped or dismounted?

Solution

This behavior, despite what it might look, is not an Oracle software problem.

Actually, when a disk is dropped or dismounted, this action is signalled to all the processes in a similar manner to the situation when a tablespace is dropped. If those processes are active, they release the file handle immediately. If the processes are idle, they have to become active to release the file handle. The thing that comes closer to such a situation is killing an idle session. The shadow process would stay forever in the system if not killed at OS level or if the client application does not attempt a new operation. As such, when an ASM disk is dropped, there would be probably more than one process still holding handles on it. The background processes, which are quite active, would quickly release the handle. The foreground processes that are active would do the same. Unfortunately, the idle processes that have an open handle, don't do anything, so they don't get to run the handle release instructions. There is very little that could be done in the Oracle software to change such a situation. These processes should either be killed at OS level or forced to perform something in the database (note that they can be ASM or main instance processes).

It sounds like given some time it will let go of the disk on its own.

I love how everyone passes the buck. RH will say it is EMC. EMC will say it's Oracle. Oracle says it's Linux. In the end, it's probably faster to just reboot than to try and figure it out.