PowerFlex: MDM switchover may cause SDS devices to go into Error state

This issue can occur with both Fine and Medium Granularity Storage Pool devices. 

SDS traces contain similar lines:   

03/09 12:50:10.732721 0x7f5315ad1db8:mosDev_TryLock:00306: MosDev try to acquire closeLock from mosDev_DoAsyncIOV for read encountered BUSY status
03/09 12:50:10.732728 0x7f5315ad1db8:contDev_UpdateDeviceError:03291: update device error devId xxxxxxxxxxxxxxxx notify needed Yes read error No write error Yes data unrecoverable No
03/09 12:50:10.732732 0x7f5315ad1db8:spefStorage_DetachInError:00921: Detach device xxxxxxxxxxxxxxxx due to device error bIsDeviceUnderAttachment=0
03/09 12:50:10.732735 0x7f5315ad1db8:spef_DetachDevice:00412: SPEF-API: Request to detach deviceDeviceId:xxxxxxxxxxxxxxxx from spef instance
03/09 12:50:10.732742 0x7f5315ad1db8:contDev_SendDeviceError:02798: Sending device error to MDM: DevId:xxxxxxxxxxxxxxxx deviceName: /dev/disk/by-id/scsi-364cd98f066b3e10024e5700ecf32a143 readError: FALSE WriteError: TRUE
03/09 12:50:10.732752 0x7f5315c05db8:spef_DetachDeviceUmtMainFunc:00496: SPEF-API: Request to detach deviceDeviceId:xxxxxxxxxxxxxxxx from spef instance a - sync started
03/09 12:50:10.732807 0x7f5315903db8:udsmLog_IssueAsyncIO:03091: ERROR: dev:xxxxxxxxxxxxxxxx, logId:1239803, Failed to issue asynchronous IO, rc: IO_HARD_ERROR (20)
03/09 12:50:10.732813 0x7f5315903db8:udsmLog_WriteBufferToDevice:04281: ERROR: rc: IO_HARD_ERROR (20)
03/09 12:50:10.732815 0x7f5315903db8:udsmLog_WriteBufferToDevice:04288: ERROR: rc: IO_HARD_ERROR (20)
03/09 12:50:10.732820 0x7f5315903db8:udsmLogSelector_DeactivateLog:00230: ERROR: rc: IO_HARD_ERROR (20)
03/09 12:50:10.732823 0x7f5315903db8:udsm_DeactivateLog:01222: ERROR: rc: IO_HARD_ERROR (20)
03/09 12:50:10.732833 0x7f5310351db8:L2PGateway_Inter_ReadL2PPage:00079: Failed to read from device , rc is:IO_HARD_ERROR
03/09 12:50:10.732837 0x7f5315903db8:spefStorage_OnDeviceError_CK:00106: Device error: IO_HARD_ERROR description: reported by SPEF layer for device xxxxxxxxxxxxxxxx
03/09 12:50:10.732840 0x7f5310351db8:L2PPageHandle_ReadToMemoryAndApplyCl:02229: Device-handle:0x7f533006bb50, DeviceId:xxxxxxxxxxxxxxxx: Failed to read PageId:8499068 to memory. rc: IO_HARD_ERROR (20)

Impact:   

  Failed devices, rebuild triggered.

A software issue can cause, under rare circumstances, some devices to go into Failed state during SDS reconfiguration (i.e. when MDM is switched over, SDS is reconnecting, device errors are cleared). There is no specific trigger for this behavior.

Clear device errors from CLI or UI.