VxFlex: SDS Host Network Instability May Cause Data Unavailability

Repeated, short-term SDS disconnections on all interfaces used for "All" or "SDS-Only" traffic, either due to network infrastructure or OS issues

Symptoms
Multiple occurrences of MDM events stating a single SDS is disconnecting and reconnecting, similar to the following:   

2018-05-31 19:22:30.656605 SDS_DISCONNECTED                 ERROR   SDS: sds05 (id: 540fde2500000005) decoupled.
2018-05-31 19:23:10.768550 SDS_IN_COOL_DOWN                 WARNING SDS: sds05 (ID 540fde2500000005) will disconnect from MDM for 15 seconds  because
2018-05-31 19:23:12.668653 SDS_DISCONNECTED                 ERROR   SDS: sds05 (id: 540fde2500000005) decoupled.
2018-05-31 19:23:50.878713 SDS_IN_COOL_DOWN                 WARNING SDS: sds05 (ID 540fde2500000005) will disconnect from MDM for 15 seconds  because
2018-05-31 19:23:52.675536 SDS_DISCONNECTED                 ERROR   SDS: sds05 (id: 540fde2500000005) decoupled.
2018-05-31 19:24:24.107139 SDS_RECONNECTED                  INFO    SDS: sds05 (ID 540fde2500000005) reconnected
2018-05-31 19:24:39.194414 SDS_IN_COOL_DOWN                 WARNING SDS: sds05 (ID 540fde2500000005) will disconnect from MDM for 15 seconds  because

Intermittent disconnection messages in SDS logs, similar to the following:   

31/05 19:38:32.468959 1e3b8eb8:contNet_OscillationNotif:01675: Con 540fde2500000005 - Oscillation of type 3 (RCV_KA_DISCONNECT) reported
31/05 19:38:32.469047 1e409eb8:contNet_OscillationNotif:01675: Con 540fde2500000005 - Oscillation of type 1 (SOCKET_DOWN) reported

Appearance of OS-level or network infrastructure-level messages and/or an increase in related counters, indicating network instability The type of messages and counters vary greatly based on the OS and infrastructure used. Contact the respective software and hardware vendors for details and detection methods.

Impact
Potential for user data unavailability The likelihood of data unavailability increases proportionally to the length of time the underlying issue persists.

When disconnections are short-lived, at certain points in time the SDS hosted on the server experiencing the issue may report one other SDS as disconnected from it, while its peers are not providing a similar report at the same exact time.

Under such circumstances, the MDM may decide to degrade parts of the data, leaving the only valid copy of those parts on the faulty host.

If the faulty host completely disconnects from the network, these parts of the data are unavailable to user applications.

• Ensure network redundancy for SDS-to-SDS network traffic (IP addresses configured as "All" or "SDS-Only").

• To avoid user data unavailability, monitor the network infrastructure for instability and either disconnect the offending ports or stop the hosted SDS, by running:   

  /opt/emc/scaleio/sds/bin/delete_service.sh

  Stopping the SDS will trigger a rebuild. When the network issue is resolved, reconnect the ports and start the SDS (if previously stopped), by running:   

  /opt/emc/scaleio/sds/bin/create_service.sh

  
   
• In case user data has become unavailable, resolve the network issue. Once resolved, the system is expected to recover automatically, making user data available.