VxRail: Add VxRail Hosts Precheck Fails with Error 'Failed to check Node compatibility'

When adding a new node to the cluster, it generates the below error:

Scenario 1: VMK0 is missing on an existing node in the cluster. 

During a node add, the VxRail Manager queries all existing nodes in the cluster. If VMK0 is missing on an existing node, it fails to respond to this query. Review lcm-web.log to determine the host that failed to respond. 

Scenario 2:  If the Cluster was upgraded from 4.5 or 4.7 to 7.x+, it may be running an old Python version. 

On VxRail 4.5.x and 4.7.x releases, the Radar tool uses python 2.7 environment. On VxRail 7.0.x releases, the Radar tool uses python 3.x environment.
When upgrading the cluster from 4.5.x or 4.7.x to 7.0.x, if any error has occurred and the upgrade is retried, it may cause the Radar tool on VxRail 7.0.x to be running with an old python 2.7 environment.

Confirm if this is the issue from VxRail Manager /var/log/microservice_log/dayone.log:

2021-10-14-03:39:41 microservice.nano-service "2021-10-14 03:39:41,162 [INFO] <Dummy-5:140386337261640> node_add_radar_check.py run_script() (176): current radar state: /mystic/radar/venv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory"

Scenario 3: Issues with ADC version 1.5.320 may cause compatibility check failures.

Confirm if this is the issue by checking the running ADC version:

cat /mystic/telemetry/DCManager/update/../conf/application.yml

/var/log/microservice_log/dayone.log shows:

"2024-01-16 11:50:09,989" microservice.nano-service "2024-01-16T11:50:09.744106811Z stdout F 2024-01-16T11:50:09.743Z <e9c5c40a861595550dbd7eebc4c94bb8> nano-service [INFO] <140063923854920> node_add_radar_check.py run_script() (197): result after join: {""total_severity"": ""CRITICAL"", ""message"": ""Plugin kgs_check_node_add required parameter(s) check_type missing.""}"

/var/log/mystic/radar.log or /var/log/mystic/radar_root/radar.log shows:

2024-01-16 12:13:14,472.472Z INFO [Radar] stdout: {
"check_id": "vxtii_check",
"result": "ERROR",
"messages": [
{
"id": "VXR304VXTI0001",
"alphaid": "vxtii_report_generation_failed",
"symptom": "Vxverify check failed with error ",
"action": "Check radar.log or vxv.log for details.",
"kb": "",
"severity": "Error",
"extension": {
"symptom_params": [
""
]
}
}
]
}
[execution_engine.py:611]

Scenario 4: Previous upgrade did not finish completely.

/var/log/microservice_log/short.term.log shows:

2021-11-19-10:01:35 microservice.nano-service "                ""status"": ""COMPLETED"","
2021-11-19-10:01:35 microservice.nano-service "                ""result"": {"
2021-11-19-10:01:35 microservice.nano-service "                    ""severity"": ""WARN"","
2021-11-19-10:01:35 microservice.nano-service "                    ""messages"": ["
2021-11-19-10:01:35 microservice.nano-service "                        {"
2021-11-19-10:01:35 microservice.nano-service "                            ""id"": ""VXR304COMVX0000"","
2021-11-19-10:01:35 microservice.nano-service "                            ""alphaid"": ""kgs_check_vxrail_system_version_inconsistent"","
2021-11-19-10:01:35 microservice.nano-service "                            ""symptom"": ""The VxRail system is in the middle of an upgrade."","
2021-11-19-10:01:35 microservice.nano-service "                            ""action"": ""Please contact Support to resolve this issue."","
2021-11-19-10:01:35 microservice.nano-service "                            ""kb"": """","
2021-11-19-10:01:35 microservice.nano-service "                            ""severity"": ""Warn"""
2021-11-19-10:01:35 microservice.nano-service "                        }"
2021-11-19-10:01:35 microservice.nano-service "                    ]"
2021-11-19-10:01:35 microservice.nano-service "                }"
2021-11-19-10:01:35 microservice.nano-service "            }"
2021-11-19-10:01:35 microservice.nano-service "        ]"
2021-11-19-10:01:35 microservice.nano-service "    }"

Scenario 1:

1. Select the host in vSphere. Go to Configure > VMkernal Adapters and create a new VMK. 
2. Place the VMK on the VxRail Management portgroup.
3. Enable the management service in Port Properties.
4. After creation, edit the newly created VMK0 and in IPv6 settings enable "Obtain IPv6 address automatically through Router Advertisement"
5. Restart services on VxRail Manager:

   systemctl restart vmware-marvin
   systemctl restart runjars

Scenario 2:

1. Confirm the cluster running code level, and download the same code level upgrade bundle from the Dell Support VxRail Product Page site:
   https://www.dell.com/support/product-details/en-us/product/vxrail-software/drivers

2. Take a snapshot of VxRail Manager.
3. Extract file Radar.tgz from the upgrade bundle, and upload the file to VxRail Manager path /home/mystic/
4. Run the below commands with a root account on the VxRail Manager to update the Radar tool:

rm -rf /mystic/radar/venv
tar -xzf /home/mystic/Radar.tgz --exclude='logs/*' --exclude='report/*' -C /mystic/radar >/dev/null
chmod -R 755 /mystic/radar
chown -R tcserver:pivotal /mystic/radar

4. Retry node addition.

Scenario 3:

If ADC is on version 1.5.320, Update ADC following VxRail: How to upgrade ADC manually.

Scenario 4:

This requires support intervention to correct baselines and VxRail Manager config service entries. Open a support ticket and reference this article.

A similar error may occur when upgrading satelite nodes where it shows a compatible host as incompatible. For information see Dell VxRail: Satellite node upgrade failed with an incompatible error