PowerFlex: Frequent MDM Disconnections

In this scenario, the Primary MDM frequently reports disconnections across various MDMs, indicating that all MDMs in the cluster are impacted.

2023-10-25 14:24:06.740 MDM_CLUSTER_NODE_DEGRADED ERROR           MDM cluster node is now DEGRADED and is in offline node node05 (ID 0411f6fe38bf3103); IPs: [14.14.14.105,15.15.15.105], Port: 9011 .
2023-10-25 14:24:06.845 MDM_CLUSTER_CONNECTED     INFO            The MDM, node05 (ID 0411f6fe38bf3103), connected after 100ms
2023-10-25 14:24:06.952 MDM_CLUSTER_NODE_NORMAL   INFO            MDM cluster node node05 (ID 0411f6fe38bf3103); IPs: [14.14.14.105,15.15.15.105], Port: 9011 is now in NORMAL state.
2023-10-25 14:24:06.952 MDM_CLUSTER_NORMAL        INFO            MDM cluster is now in NORMAL mode.
2023-10-25 14:24:42.060 MDM_CLUSTER_LOST_CONNECTION WARNING       The MDM, node02 (ID 6176009c2159e002), has lost connection to the cluster.
2023-10-25 14:24:42.060 MDM_CLUSTER_LOST_CONNECTION WARNING       The MDM, node05 (ID 0411f6fe38bf3103), has lost connection to the cluster.
2023-10-25 14:24:42.162 MDM_CLUSTER_CONNECTED     INFO            The MDM, node05 (ID 0411f6fe38bf3103), connected after 100ms
2023-10-25 14:24:42.162 MDM_CLUSTER_CONNECTED     INFO            The MDM, node02 (ID 6176009c2159e002), connected after 100ms
...

Primary MDM trc logs may report errors as follows.

2023/10/25 14:24:42.161854 7fb3d978adb0:netPath_StartAsClient_CK:00920:  :: Disconnected Live CLIENT path 0x7fb3e8113bb0 of portal 0x7fb3e8111650 net 0x7fb3e8010420 socket -1 infl
ights 0 HS:0 Start connection from conID 79b6776f71a29e00 to 0411f6fe38bf3103, to IP 15.15.15.105:9011
2023/10/25 14:24:42.161902 7fb3d978adb0:netPath_StartAsClient_CK:00920:  :: Disconnected Live CLIENT path 0x7fb3e8120b30 of portal 0x7fb3e8120930 net 0x7fb3e8010420 socket -1 infl
ights 0 HS:0 Start connection from conID 79b6776f71a29e00 to 6176009c2159e002, to IP 14.14.14.102:9011
2023/10/25 14:24:42.161919 7fb3d978adb0:netPath_StartAsClient_CK:00920:  :: Disconnected Live CLIENT path 0x7fb3e8122f90 of portal 0x7fb3e8120a30 net 0x7fb3e8010420 socket -1 infl
ights 0 HS:0 Start connection from conID 79b6776f71a29e00 to 6176009c2159e002, to IP 15.15.15.102:9011
2023/10/25 14:24:42.161934 7fb3d978adb0:netPath_StartAsClient_CK:00920:  :: Disconnected Live CLIENT path 0x7fb3e8111750 of portal 0x7fb3e8111550 net 0x7fb3e8010420 socket -1 inflights 0 HS:0 Start connection from conID 79b6776f71a29e00 to 0411f6fe38bf3103, to IP 14.14.14.105:9011

SAR data may show spikes in TCP retransmissions.

Linux 5.4.225-200.el7.x86_64                10/25/2023      _x86_64_        (64 CPU)
11:56:11 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s
12:23:38 PM      0.00      0.00      0.00      0.00      0.00
12:23:39 PM      0.00      0.00    284.00      0.00      0.00
12:23:40 PM      0.00      0.99    123.76      0.00      0.00
12:23:41 PM      0.00      1.00    501.00      0.00      0.00
12:23:42 PM      0.00      0.00    275.25      0.00      0.00
12:24:37 PM      0.00      0.00    274.00      0.00      0.00
12:24:38 PM      0.00      0.00    122.77      0.00      0.00
12:24:39 PM      0.00      0.00    276.00      0.00      0.00
12:24:40 PM      0.00      0.00    159.00      0.00      0.00
12:24:41 PM      0.00      0.00    439.00      0.00      0.00
12:24:42 PM      0.00      7.92    249.50      0.00      8.91
12:24:43 PM      0.00      0.00    109.00      0.00      2.00
12:24:44 PM      0.00      0.99    102.97      0.00      0.99
12:24:45 PM      0.00      0.00     84.00      0.00      0.00
12:24:46 PM      0.00      0.00     63.37      0.00      0.00
12:24:47 PM      0.00      0.00      0.00      0.00      0.00
12:24:48 PM      0.00      0.00    124.75      0.00      0.00
12:24:49 PM      0.00      0.00    321.00      0.00      0.00
12:24:50 PM      0.00      0.00    346.00      0.00      0.00
12:24:51 PM      0.00      0.00    316.00      0.00      0.00

There is a packet loss when pinging with jumbo frames against the data networks, which appears to be intermittent, as evidenced by the initial 12% packet loss experienced with the address 14.14.14.102, which was not replicated in subsequent tests.

[root@node01 ~]# for x in {102..105}; do echo "Testing .$x"; ping -c100 -Mdo -i .01 -s 8972 14.14.14.$x |grep -i packet; done
Testing .102
100 packets transmitted, 88 received, 12% packet loss, time 1089ms
Testing .103
100 packets transmitted, 100 received, 0% packet loss, time 1088ms
Testing .104
100 packets transmitted, 100 received, 0% packet loss, time 1088ms
Testing 105
100 packets transmitted, 100 received, 0% packet loss, time 1088ms
 
 
[root@node01 ~]# for x in (102..105); do echo "Testing .$x"; ping -c100 -Mdo -i .01 -s 8972 14.14.14.$x |grep -i packet; done
Testing .102
100 packets transmitted, 100 received, 0% packet loss, time 1089ms
Testing .103
100 packets transmitted, 100 received, 0% packet loss, time 1088ms
Testing .104
100 packets transmitted, 100 received, 0% packet loss, time 1088ms
Testing -105
100 packets transmitted, 100 received, 0% packet loss, time 1088ms

The MDM cluster reports constant Degraded states and MDMs frequently disconnecting or reconnecting. In this scenario, the intermittent packet loss or spikes in TCP retransmissions did not impact the SDS. 

Impact

The MDM cluster will report constant Degraded states and MDMs frequently disconnecting/reconnecting. In this scenario, the intermittent packet loss or spikes in TCP retransmissions did not impact the SDS. 

The network instability and intermittent packet loss appear to be related to IPv6 handling within the network. Modifying the IPv4/IPv6 address handling configuration and disabling IPv6 resolved the issue, suggesting a conflict or misconfiguration with IPv6 or that the network was more stable and efficient in operating solely on IPv4.

Run the following commands on all PowerFlex nodes:

echo ""precedence ::ffff:0:0/96 100"" >>/etc/gai.conf

echo "net.ipv6.conf.lo.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6=1" >> /etc/sysctl.conf

sed -i 's/IPv6_rpfilter=yes/IPv6_rpfilter=no/g' /etc/firewalld/firewalld.conf

echo "net.ipv6.conf.eth0.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.eth1.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.eth2.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.eth3.disable_ipv6=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.eth4.disable_ipv6=1" >> /etc/sysctl.conf

Place the node in Maintenance Mode (IMM or PMM), then reboot the node. 

Impacted Versions

Not a PowerFlex issue

Fixed In Version

While not a PowerFlex issue, PowerFlex Manager 3.8.7 and above disables IPv6 on new deployments.

See Related Article: PowerFlex: Frequent MDM Disconnects