PowerFlex 4.x PowerFlex Gateway lost connection with MDM cluster during MDM re-configuration

• MDM reconfiguration operation fails, stating cannot connect to the PowerFlex gateway.
• PowerFlex Gateway inventory operations will start failing after the MDM reconfiguration operation.
• In the Resource pane, the PowerFlex Gateway is showing "PowerFlex unreachable" or unhealthy
• The scaleio.log/scaleio_trace.log reports similar to below errors, which starts after SwitchMdmOwnership request:

2022-07-13 17:28:37,810 [https-openssl-nio-443-exec-5] DEBUG c.e.s.s.s.ConnectionServiceImpl - Send request SwitchMdmOwnership to MDM
2022-07-13 17:28:37,811 [https-openssl-nio-443-exec-5] DEBUG c.e.e.n.c.LayeredProtocolConnection - Cipher suites supported by server: [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2022-07-13 17:28:37,811 [https-openssl-nio-443-exec-5] DEBUG c.e.e.n.c.LayeredProtocolConnection - Default or requested cipher suites: [TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256]
2022-07-13 17:28:37,811 [https-openssl-nio-443-exec-5] DEBUG c.e.e.n.c.LayeredProtocolConnection - Intersection of the supported and requested cipher suites: [TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256]
2022-07-13 17:28:37,813 [https-openssl-nio-443-exec-5] DEBUG c.e.e.u.s.X509TrustManagerExtended - Trusting all the certificates
2022-07-13 17:28:37,975 [https-openssl-nio-443-exec-1-ReceiveLoop-1293774449] ERROR c.e.e.n.c.LayeredProtocolConnection - LPC-1293774449 #9dc0fdda29f79ec8 [192.168.150.26:8611]: IOException in receive loop: java.net.SocketException: Connection reset
2022-07-13 17:28:37,975 [https-openssl-nio-443-exec-1-ReceiveLoop-1293774449] INFO  c.e.e.n.c.LayeredProtocolConnection - LPC-1293774449 #9dc0fdda29f79ec8 [192.168.150.26:8611]: Attempting to re-connect
2022-07-13 17:28:37,975 [https-openssl-nio-443-exec-5] DEBUG c.e.s.s.s.ConnectionServiceImpl - Got rc SUCCESS for command SwitchMdmOwnership
2022-07-13 17:28:37,975 [command-service-0-ReceiveLoop-1271882087] ERROR c.e.e.n.c.LayeredProtocolConnection - LPC-1271882087 #858e689a28e91ed7 [192.168.150.26:8611]: IOException in receive loop: java.net.SocketException: Connection reset

To fix this issue, the user must restart the PowerFlex gateway pod.

The following command can be used for restarting the PowerFlex gateway.

kubectl delete pod -n powerflex $(kubectl get pods -A | grep block-legacy-gateway | grep -v mds | grep -i running | awk ' {print $2}')