NetWorker: VMware groups failing with error failed to load inventory from vCenter

1. NetWorker workflows associated with Group Type: VMware and has Dynamic Association enabled (Tag based association), fails with error "Failed to load inventory."
2. The workflows do not report VMs under it when an attempt is made to "Start individual clients."
3. The below errors are noticed in daemon.log:
 

nsrdisp_nwbg NSR critical Inventory session status returned with a failure. Error: 'Failed to load inventory from '<vcenter name>'. Failed to load Category/Tagging metadata: Failed to login to CIS service at 'HTTPS://<vCenter name>/rest/com/vmware/cis/session': Post https://<vcenter name>/rest/com/vmware/cis/session: net/http: request canceled (Client.Timeout exceeded while awaiting headers)'.

nsrdisp_nwbg NSR critical Inventory session status returned with a failure. Error: 'Failed to load inventory from '<vcenter name>'. Failed to load Category/Tagging metadata:invalid memory address or nil pointer dereference'

Inventory session status returned with a failure. Error: 'Failed to load inventory from '<vCenter name>'.  Failed to load Category/Tagging metadata:  Failed to send REST request to vCenter: Post https://<vCenter name>/rest/com/vmware/cis/tagging/tag-association/id:urn:vmomi:InventoryServiceTag:73f530b9-ea0b-48db-b657-c6cf308c2a53:GLOBAL?~action=list-attached-objects: net/http: request canceled (Client.Timeout exceeded while awaiting headers)'.

1. Failure from vCenter in clearing the stale tags from cs.identity:

This can be identified by running the below commands on vCenter CLI. Connect to vCenter postgres database:

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

To query the stale tag associated with VM:

# select count(*) from cis_kv_keyvalue where kv_provider like 'tagging:%' and kv_key like 'tag_association urn:vmomi:VirtualMachine:%' and regexp_replace(kv_key, 'tag_association urn:vmomi:VirtualMachine:vm-([0-9]+).*', '\1')::bigint not in (select id from vpx_vm);

To query the stale tag associated with Host: 

# select count(*) from cis_kv_keyvalue where kv_provider like 'tagging:%' and kv_key like 'tag_association urn:vmomi:HostSystem:%' and regexp_replace(kv_key, 'tag_association urn:vmomi:HostSystem:host-([0-9]+).*', '\1')::bigint not in (select id from vpx_host);

2. VAPI endpoint failing on vCenter due to memory crunch and there by dumping heap memory:

This can be identified by running the below commands on vCenter CLI:

# ls -ltrh *hprof* | awk {'print $9'}
java_pid62528.hprof
java_pid45649.hprof
java_pid36715.hprof
java_pid2514.hprof
java_pid43896.hprof
java_pid52081.hprof

Each of the above correspond to a time that you noticed the errors on daemon.log on NetWorker. 

3. Domain authentication issue.

The VMware vCenter environment has been integrated with an external authority (Active Directory/LDAP). An externally authenticated user account was created for integrating NetWorker with VMware; this account is the one use to add the VMware vCenter to NetWorker. An issue is occurring during the authentication of the external account in VMware. 

Each of the above cases must be resolved by a VMware administrator by following the below suggestions:
 

1. Stale Tags:

 To clear stale tags:

A. Stop vpxd and Content library service:

service-control --stop vmware-vpxd
service-control --stop vmware-content-library

B. Connect to vCenter postgres database:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

To delete stale tags:

delete from cis_kv_keyvalue where kv_provider like 'tagging:%' and kv_key like 'tag_association urn:vmomi:VirtualMachine:%' and regexp_replace(kv_key, 'tag_association urn:vmomi:VirtualMachine:vm-([0-9]+).*', '\1')::bigint not in (select id from vpx_vm) returning kv_key, kv_value;

delete from cis_kv_keyvalue where kv_provider like 'tagging:%' and kv_key like 'tag_association urn:vmomi:HostSystem:%' and regexp_replace(kv_key, 'tag_association urn:vmomi:HostSystem:host-([0-9]+).*', '\1')::bigint not in (select id from vpx_host) returning kv_key, kv_value;

delete from cis_kv_keyvalue where kv_provider like 'tagging:%' and kv_key like 'tag_association urn:vmomi:Datastore:%' and regexp_replace(kv_key, 'tag_association urn:vmomi:Datastore:datastore-([0-9]+).*', '\1')::bigint not in (select id from vpx_datastore) returning kv_key, kv_value;

A. Run the above two select query commands again to ensure that the count is 0.
B. Start services:

service-control --start vmware-vpxd
service-control --start vmware-content-library

C. Logout of vCenter session and login back again and validate the environment, tags, NSX, Back up, Provisioning, and so forth.

If vCenter does not look healthy, collect logs from vCenter with the command "vc-support" and contact VMware.
 

2. VAPI Crashes:

Identify the memory allocated to VAPI and increase if the memory is too low, Best practice can be determined by VMware support.
 

#cloudvm-ram-size -l | grep -i vapi

vmware-vapi-endpoint =   256

To:

#cloudvm-ram-size -l | grep -i vapi

vmware-vapi-endpoint =   1120

3. Domain authentication issue:

• NetWorker Management Console (NMC): Go to Protection->VMware View->Right-Click the vCenter->Modify vCenter.
• NetWorker Web User Interface (NWUI): Go to Protection->VMware vCenter->Select the vCenter->Click Edit.

See also:

• NetWorker: VMware View Refresh Fails "Dispatcher Request Timed Out"
• NVP vProxy: VMware View Does Not Refresh and all VM Backups Fail
• NVP vProxy: NetWorker nsrvim process runs every 15 minutes causing a high workload on vCenter Server and potential VPXD unavailability