We have four X200 nodes in our cluster running 6.5.5.11. I had to temporarily disconnect the 10Gbit interfaces on nodes #2 and #4 yesterday. I monitored our SmartConnect pools during the maintenance, and the IP addresses on those interfaces dynamically failed over to the other two nodes correctly. Once the interfaces were reconnected, I ran a manual rebalance of addresses on the pool, and the IPs rebalanced correctly.

However, I then noticed we could no longer reach the IP addresses on three of the four nodes. Currently, only IPs on node 1 can be reached. Nodes 2, 3, and 4 are all unreachable on the 10Gb interfaces. The links are green, and the IPs are assigned to them, but they are unreachable.

To troubleshoot, I created a new test pool, and moved the interfaces that weren't working into that pool with a different IP range. The IPs were successfully assigned to the interfaces, however they are still unreachable on the network.

I verified the ports on the switch are configured correctly, and verified from the switch logs that no changes occurred except that the port went down and came back up during the maintenance.

I also rebooted a node and then tested connectivity, but its still unreachable.

Node 1 is the only one working, it is also the same node that hosts the SmartConnect service IP. I'm not sure if that has anything to do with it though. This really feels as if the cluster doesn't know which IPs are on which interfaces, even though from the GUI it all looks right.

Can anyone suggest any CLI troubleshooting options? I have a ticket open but its been 12 hours and no response yet, so I'm looking for some troubleshooting ideas in the meantime. Thanks!