PowerFlex: Unexpected MDM and SDS Upgrades After Node Repurposing VOTER_ACTIVE_ATTACH_TO_OTHER_CLUSTER

- Cluster A is the affected cluster in this scenario.

- Nodes were repurposed from one Data Center to another at a different site.

- Nodes repurposed and assigned to Cluster B had MDM and SDS components installed/upgraded. Shortly afterward, Cluster A reported MDM and SDS upgrade events.

In Cluster A, the Primary MDM (PMDM) events show the MDM component on host 201 was upgraded, and 45 seconds later, the SDS component on the same host received two successive upgrades.

2025-06-30 22:45:30.898 MDM_UPGRADED              INFO          MDM upgraded to version 4.5.3000. ID: host201 (ID 3a1970dc3f839306)
2025-06-30 22:45:30.952 SDS_UPGRADED              INFO          SDS: Sds-192.x.x.18 (ID cf86cc0f00000008) upgraded to version 4.5.3000
2025-06-30 22:46:06.718 MDM_UPGRADED              INFO          MDM upgraded to version 4.5.2100. ID: host201 (ID 3a1970dc3f839306)
2025-06-30 22:46:13.031 SDS_UPGRADED              INFO          SDS: Sds-192.x.x.18 (ID cf86cc0f00000008) upgraded to version 4.5.2100

Seconds before the initial upgrade, Secondary MDM traces logged the error "VOTER_ACTIVE_ATTACH_TO_OTHER_CLUSTER" referencing cluster ID 63f8ce925f9cfb0f (Cluster B). Cluster A’s ID is a72024c2d886570f.

2025/06/30 22:45:23.597675 LOW:7f6d8075edb0:actor_NewConCB:07252: New actor connection with ID: 6ca20a9627bfb208, from 192.x.x.18: 44246
2025/06/30 22:45:23.597926 MED:7f6d80755db0:voter_GetSpammerFromAnotherCluster:04602: Adding actorID: 6ca20a9627bfb208 to spammers from other clusters array
2025/06/30 22:45:23.597931 MED:7f6d80755db0:voter_VerifyMsg:00852: [suspect spammer from INVALID] msg: [actorID: 6ca20a9627bfb208, clsUniqueID: 63f8ce925f9cfb0f, senderVoterID: 15998811151f6b0b, voterNetObjID: 2a02e46a2ddd0b0b, actorGen: 26, degradedGen: 205] local: [actorID: 324cfd6b54a07d09, clsUniqueID: a72024c2d886570f, voterID: 4c1d42b8165d040a, netObjID: 77975969213a030a, actorGen: 7, degradedGen: 183, bIsFrozen: 0, MasterHasLease: 1, CandidateHasLease: 0, ] rc: VOTER_ACTIVE_ATTACH_TO_OTHER_CLUSTER(475). Reason: Got MeMaster from actor with different cluster ID
2025/06/30 22:45:23.597939 LOW:7f6d80755db0:voter_HandleMeMaster:02553: Msg:  actorId 6ca20a9627bfb208 actorGen 26 voterId 15998811151f6b0b netObjId 2a02e46a2ddd0b0b oosIDs [] degradedGen 205 IsFrozen 0 clsID 63f8ce925f9cfb0f bMeNoMaster 0 successor 0 bHasQuorum 0 startTick 1942680772.0 senderVersion: 4.5.3000

Impact

The cluster may report DATA_DEGRADED events. Although no data unavailability occurred in this case, other scenarios could result in data unavailability being reported.

 IO retries may impact clients, resulting in timeouts. 

When nodes were repurposed into Cluster B but retained the same hostnames and IP addresses as those in Cluster A, upgrade operations performed against Cluster B were inadvertently applied to the matching nodes still present in Cluster A, resulting in unintended MDM and SDS upgrades in Cluster A.

Before repurposing nodes into any PowerFlex cluster, assign them unique hostnames and IP addresses that do not duplicate those of an existing cluster.

If nodes have already been added with conflicting identifiers, remove them from the original cluster, update their hostname and IP scheme, and then re-add them to the intended cluster.