PowerFlex: MDM Disconnections When MDM is Run on the Same Node With SDC
Summary: Primary MDM and Secondary MDM event logs report upon disconnections to the other MDM cluster members. Primary MDM failover may occur.
Symptoms
When MDM and SDC run on the same node, network issues may appear and lead to MDM disconnections.
Note: As this issue is under investigation, it is not clear why network issues appear in this case, and what is causing the network issues.
The following message type appears in the event log indicating general instability in the MDM cluster where connectivity issues and primary switch over are seen:
namdev.nsrootdev.net:2021-02-18 18:05:41.643000:0000810:MDM_CLUSTER_CONNECTED INFO The MDM,(ID 310412bb3fba5a03), connected after 460ms namdev.nsrootdev.net:2021-02-18 18:05:52.913000:0000237:MDM_CLUSTER_LOST_CONNECTION WARNING The MDM,(ID 1c04863318483100), has lost connection to the cluster. . . . namdev.nsrootdev.net:2021-02-18 18:05:53.355000:0000812:MDM_MANAGER_START INFO MDM started with the role of Manager namdev.nsrootdev.net:2021-02-18 18:05:53.466000:0000817:MDM_CLUSTER_BECOMING_MASTER WARNING This MDM, (ID 1c04863318483100), took control of the cluster and is now the Master MDM. namdev.nsrootdev.net:2021-02-18 18:05:53.469000:0000818:MDM_CLUSTER_NODE_NORMAL INFO MDM cluster node (ID 2f7796757a57ce02); IPs: [xx.xx.xxx.xx,xx.xx.xxx.xx], Port: 9011 is now in NORMAL state. namdev.nsrootdev.net:2021-02-18 18:05:53.469000:0000819:MDM_CLUSTER_NORMAL INFO MDM cluster is now in NORMAL mode. namdev.nsrootdev.net:2021-02-18 18:05:53.469000:0000820:MDM_CLUSTER_NODE_DEGRADED ERROR MDM cluster node is now DEGRADED and is in offline node naruct49siob01 (ID 5af08d0f2a534001); IPs: [xx.xx.xxx.xx,xx.xx.xxx.xx], Port: 9011 . namdev.nsrootdev.net:2021-02-18 18:05:53.509000:0000233:MDM_CLUSTER_CONNECTED INFO The MDM, (ID 1c04863318483100), connected after 300ms namdev.nsrootdev.net:2021-02-18 18:05:53.516000:0000240:MDM_CLUSTER_CONNECTED INFO The MDM, (ID 1c04863318483100), connected after 300ms namdev.nsrootdev.net:2021-02-18 18:05:53.572000:0000821:MDM_CLUSTER_NODE_NORMAL INFO MDM cluster (ID 5af08d0f2a534001); IPs: [xx.xx.xxx.xx,xx.xx.xxx.xx], Port: 9011 is now in NORMAL state. namdev.nsrootdev.net:2021-02-18 18:05:53.572000:0000822:MDM_CLUSTER_NORMAL INFO MDM cluster is now in NORMAL mode. namdev.nsrootdev.net:2021-02-18 18:05:53.572000:0000823:MDM_BECOMING_MASTER WARNING This MDM is switching to Master mode. MDM will start running.
Checking SAR (network), many retransmits (retrans/s column) are observed on the MDM nodes:
18:04:55 atmptf/s estres/s retrans/s isegerr/s orsts/s 18:05:06 175.00 2.00 780.00 0.00 1.00 18:05:08 175.00 3.00 424.00 0.00 3.00 18:05:24 171.00 0.00 517.00 0.00 0.00 18:05:39 169.61 0.00 487.25 0.00 0.00 18:05:41 144.12 6.86 429.41 0.00 0.00 18:06:11 176.24 0.00 600.00 0.00 0.00 18:06:13 170.59 0.00 414.71 0.00 0.00 18:06:49 174.00 0.00 425.00 0.00 0.00 18:07:18 174.26 0.00 506.93 0.00 0.00 18:07:53 163.00 0.00 476.00 0.00 0.00 18:07:55 173.00 0.00 595.00 0.00 0.00 18:07:57 177.00 0.00 565.00 0.00 0.00 18:08:16 174.00 0.00 607.00 0.00 0.00 18:08:18 171.00 0.00 534.00 0.00 0.00 18:08:20 163.00 0.00 657.00 0.00 0.00 18:08:25 167.31 0.00 520.19 0.00 0.00
Impact
MDM cluster is degraded, and the outcome of this state is:
- The system is in a single point of failure state if the system runs in a three-node cluster configuration.
- The system is in a single point of failure state if the system runs in a five-node cluster configuration but two MDMs get disconnected from the Primary MDM.
Note: There is no impact to the data path by running in a cluster-degraded state. However, running in a single point of failure state increases the risk factor for an MDM cluster failure, which impacts the ability of the system to serve data.
Cause
still being investigated.
Resolution
Move the MDM or the SDC process to a different node. The network issue does not appear when those two software elements are running on separated nodes.