VPLEX: Cluster Witness displaying inconsistent admin-state
Summary: VPLEX: Cluster Witness displaying inconsistent admin-state
Symptoms
Deploy and configure Cluster Witness server.
Cluster Witness displays "inconsistent" admin-state on its VPLEX CLI context after successfully deploying the server VM and configuring it in the CLI as per the applicable official SolVe Desktop procedure (depending on the installed GeoSynchrony code version):
VPlexcli:/> ll cluster-witness/
/cluster-witness:
Attributes:
Name Value
------------------ -------------
admin-state inconsistent
private-ip-address 128.221.254.3
public-ip-address X.X.X.XCluster Witness displays as "reachable" on VPN status. It seems to be properly configured and healthy:
VPlexcli:/> vpn status
Verifying the VPN status between the management servers...
IPSEC is UP
Remote Management Server at IP Address X.X.X.X is reachable
Remote Internal Gateway addresses are reachable
Verifying the VPN status between the management server and the cluster witness server...
IPSEC is UP
Cluster Witness Server at IP Address 128.221.254.3 is reachable
Cause
Inconsistent admin-state means all components are reachable over the management network but some components report their administrative state as disabled while others report it as enabled.
This is an extremely rare state.
VPlexcli:/> ll cluster-witness/components/
/cluster-witness/components:
Name ID Admin State Operational State Mgmt Connectivity
--------- -- ----------- ----------------- -----------------
cluster-1 1 enabled in-contact ok
cluster-2 2 enabled in-contact ok
server - disabled - ok <<<<Resolution
In this case, since the disabled component causing the admin-state to be "inconsistent" was the Cluster Witness server itself, enabling it with the "cluster-witness enable" command cleared the issue and the admin-state changed to "enabled":
VPlexcli:/> cluster-witness enable
VPlexcli:/> ll cluster-witness/
/cluster-witness:
Attributes:
Name Value
------------------ -------------
admin-state enabled
private-ip-address 128.221.254.3
public-ip-address X.X.X.X
Additional Information
In some rare cases the admin-state may be showing as "inconsistent" even though the VPN status is showing healthy status and you are unable to enable the cluster-witness due to having no /cluster-witness/ context showing on one of its clusters.
CLUSTER-1 VPLEX CLI:
VPlexcli:/> ll /cluster-witness/components/
/cluster-witness/components:
Name ID Admin State Operational State Mgmt Connectivity
--------- -- ----------- ----------------- -----------------
cluster-1 1 disabled - ok
cluster-2 2 disabled - ok
server - enabled unknown okVPlexcli:/> vpn status
Verifying the VPN status between the management servers...
IPSEC is UP
Remote Management Server at IP Address x.x.x.x is reachable
Remote Internal Gateway addresses are reachable
Verifying the VPN status between the management server and the cluster witness server...
IPSEC is UP
Cluster Witness Server at IP Address 128.221.254.3 is reachableVPlexcli:/> cluster-witness enable
cluster-witness enable: Evaluation of <<cluster-witness enable>> failed.
cause: Could not enable Cluster Witness.
cause: Cluster Witness cannot be enabled due to failure of a pre-check.
cause: Cluster Witness is not configured on cluster-2
CLUSTER-2 VPLEX CLI:
VPlexcli:/> ll /cluster-witness/
ls: No context found for '/cluster-witness/'VPlexcli:/> vpn status
Verifying the VPN status between the management servers...
IPSEC is UP
Remote Management Server at IP Address x.x.x.x is reachable
Remote Internal Gateway addresses are reachable
Verifying the VPN status between the management server and the cluster witness server...
IPSEC is UP
Cluster Witness Server at IP Address 128.221.254.3 is reachableVPlexcli:/> cluster-witness enable
<unknown command>: Evaluation of <<cluster-witness enable>> failed.
cause: Property 'cluster-witness' is not supported.
To resolve the issue with a cluster not having a cluster-witness context, you will run the command cluster-witness configure which should create the context if one does not already exist. Afterwards you should be able to enable the cluster-witness on the cluster like normal.
Check to see if the Cluster-Witness context exists
VPlexcli:/> ll
Name Description
------------------- ----------------------------------------------------------
clusters Contains all the clusters that are currently connected, or
have been connected during the current session.
data-migrations Contains extent migrations and device-migrations.
distributed-storage Contains distributed devices and WAN outage rule sets.
engines Contains all the engines that are currently connected.
management-server Contains the management server for the local cluster
monitoring Contains system performance monitors.
notifications Contains the call-home event-filter and its sinks.
recoverpoint Contains RecoverPoint components within this system.
system-defaults System-wide default values and settings.
If you do not see the Cluster-Witness context listed, run the command,
VPlexcli:/> cluster-witness configure
Then repeat step 1 to see if the Cluster-Witness is listed now
VPlexcli:/> ll
Name Description
------------------- ----------------------------------------------------------
cluster-witness Contains the Cluster Witness connected to all clusters <<<<
within this system.
clusters Contains all the clusters that are currently connected, or
have been connected during the current session.
data-migrations Contains extent migrations and device-migrations.
distributed-storage Contains distributed devices and WAN outage rule sets.
engines Contains all the engines that are currently connected.
management-server Contains the management server for the local cluster
monitoring Contains system performance monitors.
notifications Contains the call-home event-filter and its sinks.
recoverpoint Contains RecoverPoint components within this system.
system-defaults System-wide default values and settings.
Next check to see if cluster-witness shows as enabled by running the command,
VPlexcli:/> ll /cluster-witness/
VPlexcli:/> ll /cluster-witness/
/cluster-witness:
Attributes:
Name Value
------------------ -------------
admin-state enabled <<<<
private-ip-address 128.221.254.3
public-ip-address x.x.x.x
Contexts:
Name Description
---------- --------------------------
components Cluster Witness Components
If the admin state shows as "disabled" enable it using the command below
VPlexcli:/> cluster-witness enable
Once the admin-state shows as "enabled" check the components of the cluster-witness to ensure they all show as "enabled", as shown below.
VPlexcli:/> ll /cluster-witness/components/
/cluster-witness/components:
Name ID Admin State Operational State Mgmt Connectivity
--------- -- ----------- ------------------- -----------------
cluster-1 1 enabled in-contact ok
cluster-2 2 enabled in-contact ok
server - enabled clusters-in-contact ok
If the admin-state is still "inconsistent" after checking the state of the cluster-witness or after running the enable command, please contact VPLEX Customer Support.