VxRail: Failed to Enable Remote Connectivity with ERROR VXR034018

Symptom 1
After upgrading VxRail 8.x to 9.1.x, you may observe the following alarms in vCenter:

• VXR034018 – Failed to enable Remote Connectivity
• VXR034020 – Remote Connectivity Lost
  

For details on the cause and the available resolution steps, refer to the following Knowledge Base article: VxRail: The connection to Dell Support system is lost after upgrading VxRail 8.x to 9.1.000

Symptom 2
When enabling connectivity, the error below displays:

Figure 1. Enable Connectivity Request

The detail error log can be found in /var/log/microservice_log/short.term.log, or micro service log directly.
If VxRail Manager is running at 7.0.350: Vxm# docker service logs func_rcs-service
If VxRail Manager is running at 7.0.370 and later versions:
Vxm# kubectl logs $(kubectl get pods -o=name | grep rcs)

Many issues can cause the failure of connectivity enablement: 
 

1. Network issue between VxRail Manager and Dell Enterprise servers through ports 443 and 8443:

esrs3-core.emc.com
esrs3-coredr.emc.com

2. Wrong Secure Connect Gateway (SCG) server port number input, the port number should be 9443:

Figure 2. Port 9443

3. Access key and pin are already used.
    
4. Invalid Access Key and Pin
    
5. Universal key is expired, or node PSNT is not registered in dell backend.
    
6. The universal key is not a valid JSON format, from short.term.log:

"2023-02-09 03:45:05,333" microservice.rcs-service "2023-02-09T03:45:05.308340848Z stderr F json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"

7. No response received from Secure Remote Services, from short.term.log:

2022-08-24-16:56:57 microservice.dell-ese "8 2022-08-24 16:56:57,881 CP Server Thread-5 DEBUG DellESE.ese.srs.srswebapi LN:1164 Calls universal device key api"
2022-08-24-16:56:58 microservice.dell-ese "8 2022-08-24 16:56:58,295 CP Server Thread-5 DEBUG urllib3.connectionpool LN: 939 Starting new HTTPS connection (1): esrs3-coredr.emc.com:443"
2022-08-24-16:56:58 microservice.dell-ese "8 2022-08-24 16:56:58,612 CP Server Thread-5 WARNING ESE.CherryPy LN:1138 Web exception during getUniversalKey api call: No response received from SRS"
2022-08-24-16:56:58 microservice.rcs-service "2022-08-24 16:56:58,614 [INFO] <ThreadPoolExecutor-0_3:140391562225408> ese_service.py generate_universal_key_by_pin() (359): Generate universal key response:500,{""message"": ""Web exception during getUniversalKey api call: No response received from SRS"", ""statusCode"": 500, ""backendResponse"": {}, ""status"": ""fail""}"
2022-08-24-16:56:58 microservice.api-gateway "169.254.171.104 - - 2022-08-24T16:56:58+00:00 ""POST /rest/vxm/internal/dell-ese/v1/ese/getUniversalKey HTTP/1.1"" 500 151 """" ""python-requests/2.24.0"" """""
2022-08-24-16:56:58 microservice.dell-ese "Back-end web error: Calls universal device key api Status: 500 - "

For Issue 1: 
If the connection type is Connect Directly, VxRail Manager needs access to esrs3-coredr.emc.com and esrs3-core.emc.com, run this command from VxRail Manager:

If the connection type is Connect using Gateway Server, see KB VxRail: Next-Generation Connectivity with VxRail 7.0.350 and Later for more information.

For issue 2:
Use port number 9443 as SCG gateway port and ensure port 5700 is accessible from VxRail Manager.

For issue 3 and 4:
From Generate Access key generate new access key and PIN.

 
Figure 3. Generate Access Key

For issue 5: Contact Dell support and quote this article.

For issue 6: Contact Dell support and quote this article.

For issue 7: Contact Dell support and quote this article. 

• Exclude the failure factors such as network issue-proxy server, unique connectivity key legitimacy issue. Check the remote connector service login: /var/log/microservice_log/short.term.log or /home/mystic/ese/var/log/ESE.log.
• ESE supports two scenarios, connect backend directly or use the SCG gateway. If an issue occurred in the gateway, RSE can check the related login.