VxFlex: ESXi SDC has wrong or missing GUID/MDM IPs

A ScaleIO / VxFlex OS ESXi SDC is not connected to the cluster, when we open an SSH connection to the ESXi host, launch the bash shell and run the following command:
 

esxcli system module parameters list -m scini | head -4

We get an output like this.
 

IoctlIniGuidStr                      string  12a34b56-1234-1234-a1bc-12abc345678d  Ini Guid, for example: 12345678-90AB-CDEF-1234-567890ABCDEF    

                                  
IoctlMdmIPStr                      string  20.20.20.10,30.30.30.10              
Mdms IPs, IPs for MDM in the same cluster should be comma-separated. To configure more than one cluster use '+' to separate between IPs.For Example: 10.20.30.40,50.60.70.80+11.22.33.44 Max 1024 characters

With either incorrect or missing information



 

We can correct this by opening an SSH session to the Primary MDM node and after logging in to the ScaleIO/VxFlex OS cluster using: 

scli --login --username admin

And entering the password when prompted, we can then run the command:

scli --query_all_sdc

This gives us an output like this:

MDM restricted SDC mode: none
Query all SDC returned 5 SDC nodes.
SDC ID: eadba94600000000 Name: N/A IP: 30.30.30.14 State: Disconnected GUID: 37FB9771-D709-4873-97C3-D3B738A8B397 OS Type: ESX Loaded Version: N/A Installed Version: 2.0.11000 Restart required
    Read bandwidth:  0 IOPS 0 Bytes per-second
    Write bandwidth:  0 IOPS 0 Bytes per-second
SDC ID: eadbd05500000001 Name: N/A IP: 30.30.30.15 State: Connected GUID: A6F23F2F-CB61-4642-BB76-D5308042DCE3 OS Type: ESX Loaded Version: 2.5.0 Installed Version: 2.5.0
    Read bandwidth:  0 IOPS 0 Bytes per-second
    Write bandwidth:  0 IOPS 0 Bytes per-second
SDC ID: eadbd05600000002 Name: N/A IP: 30.30.30.12 State: Connected GUID: 12345678-90AB-CDEF-1234-567890ABCDEF OS Type: ESX Loaded Version: 2.5.0 Installed Version: 2.5.0
    Read bandwidth:  0 IOPS 0 Bytes per-second
    Write bandwidth:  0 IOPS 0 Bytes per-second
SDC ID: eadbd05700000003 Name: N/A IP: 30.30.30.13 State: Connected GUID: 18D87C16-8897-4271-B2EC-43EDE591324D OS Type: ESX Loaded Version: 2.0.11000 Installed Version: 2.0.11000
    Read bandwidth:  0 IOPS 0 Bytes per-second
    Write bandwidth:  0 IOPS 0 Bytes per-second
SDC ID: eadba94500000004 Name: N/A IP: 30.30.30.44 State: Disconnected GUID: DBAA6281-C3C5-479D-8575-7ED0DA463BD5 OS Type: LINUX Loaded Version: N/A Installed Version: 2.6.10000 Restart required
    Read bandwidth:  0 IOPS 0 Bytes per-second
    Write bandwidth:  0 IOPS 0 Bytes per-second

We can then copy the GUID from the SDC that is in a disconnected state.
Next we run: 

scli --query_cluster

And get an output like:

Cluster:
    Name: SIO_Cork, Mode: 3_node, State: Degraded, Active: 2/3, Replicas: 1/2
    Virtual IPs: 30.30.30.10
Primary MDM:
    Name: MDM2, ID: 0x0d2120467951a9b1
        IPs: 20.20.20.42, 30.30.30.42, Management IPs: 10.60.34.42, Port: 9011, Virtual IP interfaces: eth2
        Version: 2.6.10000
Secondary MDMs:
    Name: MDM1, ID: 0x241fae70512b5d10
        IPs: 20.20.20.41, 30.30.30.41, Management IPs: 10.60.34.41, Port: 9011, Virtual IP interfaces: eth1, eth2
        Status: Disconnected, Version: N/A
Tie-Breakers:
    Name: TB1, ID: 0x48bb8af4404aba92
        IPs: 20.20.20.43, 30.30.30.43, Port: 9011
        Status: Normal, Version: 2.6.10000

We will then take the virtual IPs (If available) or all the IPs and arrange them into a comma-separated list like below:

20.20.20.42,30.30.30.42,20.20.20.41,30.30.30.41
OR
30.30.30.10

We then put the gathered GUID and IPs into a command in this format. 

esxcli system module parameters set -m scini -p "IoctlIniGuidStr=<GUID> IoctlMdmIPStr=<MDM_IPs>"

esxcli system module parameters set -m scini -p "IoctlIniGuidStr=DBAA6281-C3C5-479D-8575-7ED0DA463BD5 IoctlMdmIPStr=20.20.20.42,30.30.30.42,20.20.20.41,30.30.30.41"

OR

esxcli system module parameters set -m scini -p "IoctlIniGuidStr=DBAA6281-C3C5-479D-8575-7ED0DA463BD5 IoctlMdmIPStr=30.30.30.10"

We then run this command on the ESXi host with the Disconnected SDC.

For the last step we restart the ESXi host and the SDC should reconnect when the host comes online.