Dell VxRail: vSan Node Is Showing Network Partitioned, but Ping on the vSan Network Is Successful

The following symptoms may occur:

• Node is in VSAN cluster partition
• VSAN network partition alarm is shown in Skyline health
• Node is showing network partitioned, although it can ping other nodes VSAN VMK port using command:

# vmkping -I vmk3 <VSAN_VMK_IP>

localcli vsan cluster get
Cluster Information:
   Enabled: true
   Current Local Time: 2019-03-03T07:40:03Z
   Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
   Sub-Cluster Backup UUID:
   Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740
   Sub-Cluster Membership Entry Revision: 0
   Sub-Cluster Member Count: 1
   Sub-Cluster Member UUIDs: 5b90382f-a7d7-8dc2-4358-246e96e1013a
   Sub-Cluster Membership UUID: aa767b5c-5cc5-a904-8224-246e96e1013a
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986

[root@xxxx:/tmp] esxcfg-vmknic -l | grep -i vmk3
vmk3       20336                                   IPv4      172.x.x.x                           255.xx.2xx.0   172.x.x.x    00:50:56:66:aa:ed 1500    65535     true    STATIC              defaultTcpipStack
vmk3       20336                                   IPv6      xxxx::xxxx:xxxx:xxxx:xxxx                64                     00:50:56:66:aa:ed 1500    65535     true    STATIC, PREFERRED   defaultTcpipStack

[root@xxxx:~] vmkping -I vmk3 172.x.x.x
PING 172.16.200.10 (172.16.200.10): 56 data bytes
64 bytes from 172.x.x.x   : icmp_seq=0 ttl=64 time=6.226 ms
64 bytes from 172.x.x.x   : icmp_seq=1 ttl=64 time=2.879 ms
64 bytes from 172.x.x.x   : icmp_seq=2 ttl=64 time=0.134 ms
 
--- 172.x.x.x ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.134/3.080/6.226 ms
[root@xxxxx:~] vmkping -I vmk3 172.xx.xx.11
PING 172.x.x.x  (172.x.x.x   ): 56 data bytes
64 bytes from 172.x.x.x: icmp_seq=0 ttl=64 time=0.200 ms
64 bytes from 172.x.x.x: icmp_seq=1 ttl=64 time=0.163 ms
64 bytes from 172.x.x.x: icmp_seq=2 ttl=64 time=0.152 ms
 
--- 172.x.x.x ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.152/0.172/0.200 ms

• Unicast agent list is invalid or incomplete.
• One or more hosts cannot communicate with other VSAN hosts in the cluster. 

1. Run this command on all hosts before changing the unicast agent list.

esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates

2. Add the missing unicast agent entries.
 

esxcli vsan cluster unicastagent add -t node -u <uuid> -U <unicastSupport> -a <ip> -p <port> -i <interface>

Example:
 

esxcli vsan cluster unicastagent add -t node -u 57ea62b0-9df6-6532-110c-9457a55a37d0 -U true -a XX.XXX.XXX.XX-p 12321 -i vmk3
 

[root@xxxxx:~] esxcli vsan cluster unicastagent list
NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name
------------------------------------  ---------  ----------------  -------------  -----  ----------
5b90378f-0b0b-2da4-933b-246e96e101b6          0              true  172.x.x.x  12321
5b90380e-927b-73ce-159f-246e96e1018a          0              true  172.x.x.x  12321
[root@xxxx:~]
 
[root@xxxxxx:~] esxcli vsan cluster unicastagent list
NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name
------------------------------------  ---------  ----------------  -------------  -----  ----------
5b90378f-0b0b-2da4-933b-246e96e101b6          0              true  172.x.x.x 12321
5b90382f-a7d7-8dc2-4358-246e96e1013a          0              true  172.x.x.x 12321

[root@xxxxx:~]
 
[root@xxxx:/tmp] esxcli vsan cluster unicastagent list
NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name
------------------------------------  ---------  ----------------  -------------  -----  ----------
5b90380e-927b-73ce-159f-246e96e1018a          0              true  172.x.x.x  12321
5b90382f-a7d7-8dc2-4358-246e96e1013a          0              true  172.x.x.x  12321
[root@xxxxx:/tmp]
 
 
[root@xxxx:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2019-03-03T07:48:55Z
   Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 5b90378f-0b0b-2da4-933b-246e96e101b6
   Sub-Cluster Backup UUID: 5b90380e-927b-73ce-159f-246e96e1018a
   Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740
   Sub-Cluster Membership Entry Revision: 3
   Sub-Cluster Member Count: 3
   Sub-Cluster Member UUIDs: 5b90378f-0b0b-2da4-933b-246e96e101b6, 5b90380e-927b-73ce-159f-246e96e1018a, 5b90382f-a7d7-8dc2-4358-246e96e1013a
   Sub-Cluster Membership UUID: d519d05b-a227-5621-66dc-246e96e101b6
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986

 

3. Once the unicast agent list has been fixed on all hosts, run the below command on all hosts to set IgnoreClusterMemberListupdates back to its default setting of 0.

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates 

See official Broadcom KB Network partition caused by an invalid/incomplete unicast agent list on vSAN host(s)