Avamar - Dell EMC Cloud Director Data Protection Extension (DPE) new deployment fails with 'The cloud credentials are invalid.' error

During initial deployment of Dell EMC Cloud Director Data Protection Extension (DPE)  version 19.3 or higher the initial node deployment step is failing on for vCloud  10.x environment 

Output looks like this

dpe.example.com: # vcp-cli node init -c=node-init.yaml
Input password:
Error:
The cloud credentials are invalid.

However the vcloud credentials  in the node-init.yaml are correct and login into vmware provider ui  (https://vcloud.example.com/provider ) in a browser are successful.


On the DPE node in the /var/log/vcp/vcp-manager.log we see the following 
 

2021-04-22 14:47:04.218  INFO 12941 --- [https-jsse-nio-9000-exec-3] com.emc.vcp.dpem.service.NodeService     : login to cloud: vcloud.example.com by user administrator@system
2021-04-22 14:47:04.243  WARN 12941 --- [https-jsse-nio-9000-exec-3] c.e.v.d.e.ExceptionConfiguration         :
com.emc.vcp.dpem.exception.AppServerException: The cloud credentials are invalid.
        at com.emc.vcp.dpem.service.NodeService.cloudLogin(NodeService.java:391)
        at com.emc.vcp.dpem.service.NodeService.setCloud(NodeService.java:355)
        at com.emc.vcp.dpem.service.NodeService.init(NodeService.java:203)
        at com.emc.vcp.dpem.controller.NodeController.init(NodeController.java:50)
        at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 

The vmware cloud system administrator disabled the legacy api endpoint. 

Normally this service is enabled by default and is required for DPE.     In vcloud cell the vmware administrator ran this vcd cell management command  to change the  API behavior,  this vmware command will cause DPE deployment failure. 
 

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.api.legacy.nonprovideronly -v true

More information see this page:  https://docs.vmware.com/en/VMware-Cloud-Director/10.0/com.vmware.vcloud.install.doc/GUID-84390C8F-E8C5-4137-A1A5-53EC27FE0024.html

Troubleshooting: 

Following vmware kb 56948   we can see that even to api/sessions  endpoint also fail with HTTP 401 regardless of which API version or username/password is used. 

For example this curl command will test login to api/sessions using 30.0  username administrator@system/password 

curl -k -I --header "Accept: application/*;version=30.0"  --header "Authorization: Basic `echo -n administrator@system:Password |base64`"  --request POST https://vcloud.example.com/api/sessions

When legacy API is enabled the curl command should return  HTTP 200 status code  and auth token like this 
 

HTTP/1.1 200 OK
Date: Thu, 03 Jun 2021 19:59:20 GMT
X-VMWARE-VCLOUD-REQUEST-ID: c8cb4fe3-e9ea-4abc-a135-cb284b7e1ea5
X-VMWARE-VCLOUD-ACCESS-TOKEN: eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbmlzdHJhdG9yIiwiaXNzIjoiYTkzYzlkYjktNzQ3MS0zMTkyLThkMDktYThmN2VlZGE4NWY5QGNhODM3MGRmLWFhM2QtNGE4YS1iZDNkLTM5MzZjOGJkMzAxNiIsImV4cCI6MTYyMjgzNjc2MCwidmVyc2lvbiI6InZjbG91ZF8xLjAiLCJqdGkiOiIyYjIyZDkwNDk1NTI0MDJjOTI4YzRmYzFiYzUyNGY5YiJ9.HttOVBg-mRm94c64GdTyvofcK6m22PSPbY-qNrKnVGg6ATcfabRsSJ_q-3YdKGx37BizRGOnxGwCG3mVc3O4suv6qU8WOQgmhd9-f-FF-xL9KHz3m5t2dxwSIUOJ0-0ePLfLrYdBy5l-H-1BHaU9bsMnGXem9ql1oAAuVAf5Epv0lZ7j9CBQG7-V1HP35dv2FAUuS5Mdppc1onOlOhxsBU8N9DKzy04wWkRAXpOz1z32lE4vKP-gqlw8nNxI6z_XpaUcg-BWuduhJt2jJrqTKyEo1lqvTixGyNIUI7XSJ1JDP76YgN1jO8qcl4NsYihRe60X8Fqv-z7xlcA_9j1Rpg
X-VMWARE-VCLOUD-TOKEN-TYPE: Bearer
x-vcloud-authorization: 2b22d9049552402c928c4fc1bc524f9b
Content-Type: application/vnd.vmware.vcloud.session+xml;version=30.0
X-VMWARE-VCLOUD-REQUEST-EXECUTION-TIME: 404
Cache-Control: no-store, must-revalidate
Vary: Accept-Encoding, User-Agent
Content-Length: 2512
dpe-test:~ #

when  api/sessions   is disabled  (vcloud.api.legacy.nonprovideronly -v true )   the HTTP respond will always be HTTP 401 regardless of the username and password.  

dpe-test:~ # curl -k -I --header "Accept: application/*;version=30.0"  --header "Authorization: Basic `echo -n administrator@system:Password |base64`"  --request POST https://vcloud.example.com/api/sessions
HTTP/1.1 401 Unauthorized
Date: Thu, 03 Jun 2021 21:08:26 GMT
X-VMWARE-VCLOUD-REQUEST-ID: f620175f-0df2-43dd-8ac6-0b8246ca9e2f
Content-Length: 0

In order to use Dell EMC Cloud Director Data Protection Extension (DPE)  we will require the vcloud system administer to re-enabled the legacy  api  endpoint by running this command on vcloud directory cells.  

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.api.legacy.nonprovideronly -v false