VPLEX: Unable to communicate with Cluster Witness Server. (User Correctable)
Summary: VPLEX: Unable to communicate with Cluster Witness Server. (User Correctable)
Symptoms
Cluster Witness Server is in an unknown state.
VPlexcli:/> ll /cluster-witness/** /cluster-witness: Attributes: Name Value ------------------ ------------- admin-state unknown private-ip-address 128.221.254.3 public-ip-address xx.xx.xx.65 Contexts: Name Description ---------- -------------------------- components Cluster Witness Components /cluster-witness/components: Name ID Admin State Operational State Mgmt Connectivity ----------------- -- ----------- ----------------- ----------------- cluster-1 1 disabled - ok cluster-2 2 disabled - ok server - unknown - failed /cluster-witness/components/cluster-1: Name Value ----------------------- ------------------------------------------------------ admin-state disabled diagnostic INFO: Cluster Witness is not enabled on cluster-1, so no diagnostic information is available id 1 management-connectivity ok operational-state - /cluster-witness/components/cluster-2: Name Value ----------------------- ------------------------------------------------------ admin-state disabled diagnostic INFO: Cluster Witness is not enabled on cluster-2, so no diagnostic information is available id 2 management-connectivity ok operational-state - /cluster-witness/components/server: Name Value ----------------------- ------------------------------------------------------ admin-state unknown diagnostic WARNING: Cannot establish connectivity with Cluster Witness Server to query diagnostic information. id - management-connectivity failed operational-state -
Cluster-witness enable fails with error "Unable to communicate with Cluster Witness Server. Check the state of the Cluster Witness Server and its connectivity and try again."
VPlexcli:/> cluster-witness enable cluster-witness enable: Evaluation of <<cluster-witness enable>> failed. cause: Could not enable Cluster Witness. cause: Cluster Witness cannot be enabled due to failure of a pre-check. cause: Unable to communicate with Cluster Witness Server. Please check the state of the Cluster Witness Server and its connectivity and try again.
VPN status shows the correct information about one cluster but does not show the Cluster Witness Server (CWS) VPN information about another cluster:
cluster-1:
VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address xx.xx.xx.140 is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable
cluster-2:
VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address xx.xx.xx.78 is reachable Remote Internal Gateway addresses are reachable
No information related to CWS was reported.
Cause
The output of /etc/ipsec.conf on cluster-1 Management Server is:
service@cluster-1:/etc> cat /etc/ipsec.conf # Add connections here. # Setup a tunnel between the management servers and their networks # "left" means local, "right" means remote. # Connection between Cluster Witness Server and Management Server conn net-witness type=tunnel keyexchange=ikev2 mobike=no left=%defaultroute leftsubnet=128.221.252.32/27,128.221.253.32/27 leftcert=hostCert.pem right=xx.xx.xx.65 rightsubnet=128.221.254.3/32 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN CWS, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start # Connection between Management Server 1 and Management Server 2 conn net-net type=tunnel keyexchange=ikev2 mobike=no left=%defaultroute leftsubnet=128.221.252.32/27,128.221.253.32/27 leftcert=hostCert.pem right=xx.xx.xx.140 rightsubnet=128.221.252.64/27,128.221.253.64/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKMxxxxxxxxxxy, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start
Whilst the output of /etc/ipsec.conf in the cluster-2 Management server is:
service@cluster-2:/etc> cat /etc/ipsec.conf # Add connections here. # Setup a tunnel between the management servers and their networks # "left" means local, "right" means remote. conn net-net type=tunnel keyexchange=ikev2 mobike=no left=%defaultroute leftsubnet=128.221.252.64/27,128.221.253.64/27 leftcert=hostCert.pem right=xx.xx.xx.78 rightsubnet=128.221.252.32/27,128.221.253.32/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKMxxxxxxxxxxx, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start
It shows no information related to the VPN between CWS and cluster-2.
Resolution
VPlexcli:/> configuration cw-vpn-configure -i xx.xx.xx.65
After doing it, /ect/ipsec.conf shows the CWS VPN information, VPN status shows it as well and cluster-witness shows no errors and can be enabled/disabled without issues.
VPlexcli:/> ll /cluster-witness/* /cluster-witness/components: Name ID Admin State Operational State Mgmt Connectivity ----------------- -- ----------- ------------------- ----------------- cluster-1 1 enabled in-contact ok cluster-2 2 enabled in-contact ok server - enabled clusters-in-contact ok