powermt commands fail with "ERROR: Device Lam claim failed"

This particular issue can only happen when the server is attached to both PowerPath managed and PowerPath non-managed arrays and following configuration changes on both arrays. In our example, some Clariion devices had been removed but the corresponding pseudo device had not been cleaned up. Then IBM storage was added and the hdisks, initially used as a path to these removed Clariion devices, were re-used to describe the new IBM disks. This resulted into these incorrect entries in the ODM.   

Environment :
OS : AIX (any flavour)
DELL EMC SW : PowerPath for AIX (any release)
non DELL EMC HW : disks from an array not manageable by PowerPath.

In an AIX environment, powermt commands fail with "ERROR: Device Lam claim failed". This error is not documented in our Dell EMC PowerPath Family CLI and System Messages Reference.

The message is reported by Powerpath when PowerPath cannot determine which LAM (Loadable Array Module) a device belongs to. Because the PowerPath commands are not working, troubleshooting can only be done looking at the ODM. If Powerpath is looking for a LAM and can't find the appropriate one, this means that there is a pseudodevice, with entries in the ODM, pointing to a type of device not expected by PowerPath.

In the case which caused the creation of this article, we had the following in the ODM :

CuAt:
name = "hdiskpower29"
attribute = "vpd_map"
value = "MF0808C,TM1010C,RL2004C,Z00008X,Z1040780C,SN081083X"
type = "V"
generic = ""
rep = "sl"
nls_index = 0 

Checking the PdAt for this kind of device, we find :

PdAt:
uniquetype = "disk/fcp/2145"
attribute = "vpd_map"
deflt = "MF0808C,TM1010C,RL2004C,Z00008X,Z1040780C,SN081083X"
values = ""
width = ""
type = "V"
generic = ""
rep = "sl"
nls_index = 0

This is not a DELL EMC type of disk nor anything managed by PowerPath : this is an IBM disk.

Easier evidence of the nature of the device is given by the unique_id attribute of the pseudo device. In our case, we had (the real value was changed to preserve confidentiality) :

CuAt:
    name = "hdiskpower29"
    attribute = "unique_id"
    value = "33213600507680C80017D3800000000000XXXXXXX4503IBMfcp"
    type = "R"
    generic = ""
    rep = "s"
    nls_index = 0
 
When such entries exist in the ODM, and when the pseudo device (here hdiskpower29) is referenced in"powermt_custom.xml", then "powermt config" will not be able to associate a LAM to the device and will fail.and the other "powermt" command will fail with the same error.

Because all the "powermt" commands fail, it is not possible to update the "powermt_custom.xml" file with a "powermt save". Deleting the "powermt_custom.xml" file would be a very bad idea because of the risk of loosing the relationship between the correct pseudo devices and the PowerPath managed array devices.

The only way to clear the issue is to remove from the ODM all the pseudo devices where the "unique_id" attribute is XXXXIBMfcp. And rather than using dangerous "odmdelete" commands, it is advised to simply use a "rmdev -dl <pseudo_device>" command.

Here is an example of the commands which were run to remove the incorrect entries in our example (of course, every case is unique and the below list is just an example) :

for i in 29 30 31 32 33 34 39 40 41 42 43 44 45 46 47 48 125 136 137 138 167 168 169 170 171 172 173 216 217 267 522 523 524 525 526 527
do
rmdev -dl hdiskpower$i
done

Following this clean-up, run a "powermt config" followed by a "powermt save". At this stage, you might also discover some "dead" paths (for pseudo devices also removed from the configuration but where the corresponding hdisks have not been reused to point to non PowerPath managed disks and which were not spotted in the ODM when checking the "unique_id" attribute) in "powermt display". These entries will be cleaned up with a "powermt check" command. In such a case, do not forget to run a "powermt save" again after this final clean-up.