PowerProtect DP Series and IDPA: There Are Currently 2 Usable Fault Domains. The Operation Requires 1 More Usable Fault Domains

An insufficient number of Fault Domains are currently present in the environment for component placement.

IDPA uses "vSAN Default Storage Policy" which means if one node has issues snapshots won't be able to be created by Avamar.

One more hosts on vSAN are offline. 
One more hosts on vSAN are in maintenance mode or decommissioned state.

Because of this, objects will go red which is an indication of a none healthily node, you can run #esxcli vsan health cluster list -w





 

 

1. Log into VCSA and Check for any none responding ESXi.

If you found any create a tunnel to the correct iDRAC (https://support.emc.com/kb/524395) and check for HW failures, if the node is in a hung state you will need to collect TSR logs and power cycle (cold boot) from iDRAC UI.

Once the node is back online you will need to reconfigure for vSphere HA from VCSA

 



2. Log into VCSA and Check for any ESXi showing into maintenance mode.

Right click on it - Go to Maintenance Mode - Exit Maintenance Mode




3. ESXi node into Decom State.

To resolve this, run the below command on any ESXi and make sure there are no hosts in Decom State. Anything other than 0 means there is a host in decommissioned mode.

Example below shows node # 2 in decommissioned mode.

for i in $( cmmds-tool find -f json -t HOSTNAME | grep -iE "uuid" | awk '{print $2}'| sed "s/\"//g" |sed "s/\,//g"); do cmmds-tool find -u $i -f json | grep -E "decom|hostname" | awk '{print $2,$3}' ;done | sed 'N;s/\n/ /'

[root@esx2-5800-crk:~] for i in $( cmmds-tool find -f json -t HOSTNAME | grep -iE "uuid" | awk '{print $2}'| sed "s/\"//g" |sed "s/\,//g"); do cmmds-tool find -u $i -f json | grep -E "deco
m|hostname" | awk '{print $2,$3}' ;done | sed 'N;s/\n/ /'
{"decomState": 6, {"hostname": "esx2-5800-crk.dp.ce.gslabs.lab.emc.com"},
{"decomState": 0, {"hostname": "esx1-5800-crk.dp.ce.gslabs.lab.emc.com"},
{"decomState": 0, {"hostname": "esx3-5800-crk.dp.ce.gslabs.lab.emc.com"},

To resolve this log into ESXi SSH using root and run folllowing command:

#localcli vsan maintenancemode cancel

Note: Most of the time the node will be in decommissioned mode, but VCSA will not show the node in any bad state.