PowerFlex 3.X: Add SDC command causing MDM panic and failover

1.MDM is running in "Restricted mode"
2.User attempts to add an SDC with IPs that don't exist in the approved SDC IPs - which forces the MDM to create a new SDC object
3.SDC name that is used is of SDC name which already existed before
4.MDM panic occurs on the primary MDM and the secondary MDM takeover



1.The user executes add SDC command using IPs and not GUID, and receives communication error as stdout for example: 

# scli --add_sdc --sdc_ip 123.234.234.201 --sdc_name SDC40
Error: MDM failed command.  Status: Communication error

2.Checking MDM exp.0 file, the panic below could be observed

21/12 02:43:26.897784 Panic in file /data/builds/workspace/ScaleIO-Common-Job@2/src/mdm/control/obj_container.c, line 2291, function objContainer_GetObjId, PID 30902.Panic Expression (((void *)0) != (pObjHeader)) && ((pObjHeader)->magic == 0x68cab8db) .
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(mosDbg_PanicPrepare+0x13a) [0x9a51ca]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(objContainer_GetObjId+0xe8) [0x981768]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(iniMgr_ApproveInitiator+0x359) [0x8c7fe9]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175() [0x562a08]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(netRecvGroup_WaitForWork+0x3dc) [0x7b2cfc]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(netRecvGroup_WaitForWorkLoop+0x18) [0x7b3008]
/opt/emc/scaleio/mdm/bin/mdm-3.5.1000.175(mosUmt_StartFunc+0x7a) [0x7fb5da]
/lib64/libc.so.6(+0x48140) [0x7ff1b5a8d140]
[(nil)]

 Impact:
1.Add SDC command will fail - SDC won't be added to the system 
2.MDM failover

Before PowerFlex v3.0, the MDM component stored all SDCs IPs, even for disconnected SDCs. From version 3.0 and on, the MDM component tracks only connected or approved SDC IPs.

As part of the "add SDC" command, the user has to use additional parameters such as: "SDC IPs" or "SDC GUID" with an optional parameter of "SDC name".  
Once the "add SDC" command is issued, the MDM will try to lookup for an existing SDC with those parameters, in our scenario, the user supplied an unfamiliar "SDC IPs", and accordingly, 
the MDM is not familiar with the newly supplied "SDC IPs" due to the change introduced in version 3.0, hence MDM has created a new SDC object.

In addition, the "SDC name" parameter that was used while executing the command was an existing SDC name, hence it has immediately deleted the SDC object that was just created as MDM has detected that this name is already in use.
Once MDM tried to access the deleted SDC object, it has asserted and a failover occurred.
 

To overcome this scenario and successfully add SDCs which were connected previously, please use the "SDC GUID" parameter instead of "SDC IP". 
For example: 

scli --add_sdc --sdc_guid FA4BFBF6-546A-11E8-B40D-0050568D283E

Impacted Versions

All versions higher than 3.0