powermt commands fail with "ERROR: Device Lam claim failed"

This particular issue can only happen when the server is attached to both PowerPath managed and PowerPath non-managed arrays and following configuration changes on both arrays. In our example, some Clariion devices had been removed but the corresponding pseudo device had not been cleaned up. Then IBM storage was added and the hdisks, initially used as a path to these removed Clariion devices, were reused to describe the new IBM disks. This resulted into these incorrect entries in the ODM

Environment:
OS: AIX (any flavour)
DELL SW: PowerPath for AIX (any release)
Non-DELL HW: disks from an array not manageable by PowerPath.

In an AIX environment, powermt commands fail with "ERROR: Device Lam claim failed." This error is not documented in our Dell PowerPath Family CLI and System Messages Reference.

PowerPath reports this message when PowerPath cannot determine which Loadable Array Module (LAM) a device belongs to. Because the PowerPath commands are not working, troubleshooting can only be done looking at the ODM. If PowerPath is looking for a LAM and cannot find the appropriate LAM, this means that there is a pseudodevice, with entries in the ODM, pointing to a type of device not expected by PowerPath.

In the case which caused the creation of this article, we had the following in the ODM:

CuAt:
name = "hdiskpower29"
attribute = "vpd_map"
value = "MF0808C,TM1010C,RL2004C,Z00008X,Z1040780C,SN081083X"
type = "V"
generic = ""
rep = "sl"
nls_index = 0

Checking the PdAt for this kind of device, we find:

PdAt:
uniquetype = "disk/fcp/2145"
attribute = "vpd_map"
deflt = "MF0808C,TM1010C,RL2004C,Z00008X,Z1040780C,SN081083X"
values = ""
width = ""
type = "V"
generic = ""
rep = "sl"
nls_index = 0

This is not a DELL type of disk nor anything managed by PowerPath: this is an IBM disk.

The unique_id attribute of the pseudo device gives easier evidence of the nature of the device. In our case, we had (the real value was changed to preserve confidentiality):

CuAt:
    name = "hdiskpower29"
    attribute = "unique_id"
    value = "33213600507680C80017D3800000000000XXXXXXX4503IBMfcp"
    type = "R"
    generic = ""
    rep = "s"
    nls_index = 0

When such entries exist in the ODM, and when the pseudo device (here hdiskpower29) is referenced in "powermt_custom.xml," then "powermt config" cannot associate a LAM to the device and fail, and the other "powermt" command fail with the same error.

Because all the "powermt" commands fail, it is not possible to update the "powermt_custom.xml" file with a "powermt save." Deleting the "powermt_custom.xml" file would be a bad idea because of the risk of loosing the relationship between the correct pseudo devices and the PowerPath managed array devices.

The only way to clear the issue is to remove from the ODM all the pseudo devices where the "unique_id" attribute is XXXXIBMfcp. And rather than using dangerous "odmdelete" commands, it is advised to simply use a "rmdev -dl <pseudo_device>" command.

Here is an example of the commands which were run to remove the incorrect entries in our example (every case is unique and the below list is an example):

for i in 29 30 31 32 33 34 39 40 41 42 43 44 45 46 47 48 125 136 137 138 167 168 169 170 171 172 173 216 217 267 522 523 524 525 526 527
do
rmdev -dl hdiskpower$i
done

Following this clean-up, run a "powermt config" followed by a "powermt save." At this stage, you might also discover some "dead" paths (for pseudo devices also removed from the configuration but where the corresponding hdisks have not been reused to point to non PowerPath managed disks and which were not spotted in the ODM when checking the "unique_id" attribute) in "powermt display." These entries are cleaned up with a "powermt check" command. In such a case, do not forget to run a "powermt save" again after this final clean-up.