VxRail: Nic Configuration Page Shows HTTP Failure Response Error During Node Expansion

NIC Configuration page shows http failure response error during node expansion.

Short.term.log:

2022-07-04-01:03:31 microservice.ms-day2 "2022-07-04 01:03:31,095 [ERROR] <ThreadPoolExecutor-0_49:140246234617600> exp_general_controller.py get() (108): Internal Error During discovering configured hosts pnic info."
2022-07-04-01:03:31 microservice.ms-day2 "Traceback (most recent call last):"
2022-07-04-01:03:31 microservice.ms-day2 "  File ""/home/app/runtime_operations_app/main/controllers/exp_general_controller.py"", line 100, in get"
2022-07-04-01:03:31 microservice.ms-day2 "    hosts_pnic_info = NodeNicService().query_configured_hosts_pnic_info(segment_label)"
2022-07-04-01:03:31 microservice.ms-day2 "  File ""/home/app/runtime_operations_app/main/services/node_nic_service.py"", line 157, in query_configured_hosts_pnic_info"
2022-07-04-01:03:31 microservice.ms-day2 "    config_hosts = self._assemble_configured_hosts_pnic_info(configured_hosts_pnic_info)"
2022-07-04-01:03:31 microservice.ms-day2 "  File ""/home/app/runtime_operations_app/main/services/node_nic_service.py"", line 124, in _assemble_configured_hosts_pnic_info"
2022-07-04-01:03:31 microservice.ms-day2 "    self._update_configured_hosts_pnic_vds_info(config_hosts)"
2022-07-04-01:03:31 microservice.ms-day2 "  File ""/home/app/runtime_operations_app/main/services/node_nic_service.py"", line 87, in _update_configured_hosts_pnic_vds_info"
2022-07-04-01:03:31 microservice.ms-day2 "    cluster_host = list(filter(lambda host:host.hostname == config_host['hostname'], cluster_hosts))[0]"
2022-07-04-01:03:31 microservice.ms-day2 "IndexError: list index out of range"

There are two different causes for this issue:

1. The DVS name of one of the nodes is different with other nodes in the same cluster. This causes a cluster information mismatch and fails to show the NIC configuration information in the page. 
2. The DVS name different from host and network part of GQL.
    

Cause 1:  If this issue occurred, run the below command on VxRail Manager to obtain all the node's information. This command displays all the cluster information in JSON format.

curl -k -H "Content-Type: application/json" -X POST -u <username>:<Password> --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -d '{"query":"query {cluster(moid : \"<mob-ID>\") {host{name hardware{systemInfo{serialNumber}}config{virtualNicManagerInfo{netConfig{nicType selectedVnic}}network{proxySwitch{dvsUuid dvsName hostLag{lagKey lagName uplinkPort{key value}}spec{backing{pnicSpec{pnicDevice uplinkPortKey uplinkPortgroupKey}}}uplinkPort{key value}}}}configManager{networkSystem{networkInfo{netStackInstance{name ipRouteConfig{defaultGateway}}}networkConfig{vnic{device spec{ip{ipAddress subnetMask}distributedVirtualPort{portgroupKey}}}}}}} network{moid name backingType config{key type defaultPortConfig{vlan{vlanId}uplinkTeamingPolicy{uplinkPortOrder{activeUplinkPort standbyUplinkPort inherited}}}distributedVirtualSwitch{name moid}}}}}", "variables":null,"operationName":null}' http://127.0.0.1/rest/vxm/internal/do/v1/network/vds/query

In the command, replace the value with the below interpretation​​​​​.

The value of <username>, <Password>, <mob-ID> should be replaced based on the running cluster information.
The "<mob-ID>" is cluster MoRef ID: vc_mob/content/rootFolder/childEntity/hostFolder/childEntity(domain-xx)
The "<username>, <Password>" is VxRail Manager's root credential

​​​​​Filter the contents of the output and compare the dvsName value of the hosts to identify the host with different DVS name with other hosts. (Even the dvsUuid values of the hosts are the same)  

Correct the DVS name of the host in vCenter.
 

Cause 2: Run the following command and confirm both of the output DVS Name are same.

curl -s -k --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -H "Content-Type: application/json" \
-d "$(cluster_moid=$(curl -s --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock \
http://127.0.0.1/rest/vxm/internal/configservice/v1/configuration/keys/cluster_moid | jq -r .value); \
printf '{"query":"query($moid:ID!){cluster(moid: $moid) { host { hardware { systemInfo { serialNumber } } config { network { proxySwitch { dvsUuid dvsName } } } } network { moid config { distributedVirtualSwitch { name moid } } } }}","variables":{"moid":"%s"}}' "$cluster_moid")" \
http://127.0.0.1/rest/vxm/internal/do/v1/network/vds/query | jq |grep name

curl -s -k --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -H "Content-Type: application/json" \
-d "$(cluster_moid=$(curl -s --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock \
http://127.0.0.1/rest/vxm/internal/configservice/v1/configuration/keys/cluster_moid | jq -r .value); \
printf '{"query":"query($moid:ID!){cluster(moid: $moid) { host { hardware { systemInfo { serialNumber } } config { network { proxySwitch { dvsUuid dvsName } } } } network { moid config { distributedVirtualSwitch { name moid } } } }}","variables":{"moid":"%s"}}' "$cluster_moid")" \
http://127.0.0.1/rest/vxm/internal/do/v1/network/vds/query | jq |grep dvsName