We have Scaleio version 2.0.5014 installed and configured in 3-host mode on CentOS 7.

It worked fine for 2 month.

Today morning 2 of 3 SDS (#1 and #2) lost connection and stay "Disconnected".

Reboot is not helpfull.

The network is working without problems, ping passes between all hosts.

Test sds gives error:

[root@scaleio1 cfg]# scli --start_sds_network_test --sds_ip 192.168.201.24

Error: MDM failed command.  Status: SDS is being configured. Please retry the command.

But "scli --query_sds --sds_ip 192.168.201.24" working:

[root@scaleio1 cfg]# scli --query_all_sds

Query-all-SDS returned 3 SDS nodes.

Protection Domain 4ed2eb6300000000 Name: Applico

SDS ID: a5361e3300000002 Name: SDS_192.168.201.24 State: Disconnected, Join-Pending IP: 192.168.201.24 Port: 7072 Version: 2.0.5014

SDS ID: a5361e3200000001 Name: SDS_192.168.201.14 State: Disconnected, Join-Pending IP: 192.168.201.14,192.168.202.12 Port: 7072 Version: 2.0.5014

SDS ID: a5361e3100000000 Name: SDS_192.168.201.34 State: Connected, Joined IP: 192.168.201.34,192.168.202.32 Port: 7072 Version: 2.0.5014

"scli --query_sds_connectivity_status --protection_domain_name Applico" hangs with no response.

What can we do to resolve this issue?