VCF on VxRail: Upgrade failed on node due to NSX-T VIB version error

Summary: VxRail Upgrade failed on node due to NSX-T VIB version error.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms

VCF on VxRail upgrade failure when it reaches VxRail manager upgrade portion.

Source version: VCF on VxRail 4.5.2
Target version: VCF on VxRail 5.0

Failure message from SDDC Manager:
VxRail component upgrade failed with error [vxrail updateUpgradeProgress] VxRail Upgrade returned status as Failure : Failure occurred while running an upgrade for bundle: D1JM1_VxRail-8.0.200-Composite-Upgrade-Slim-Package-for-7.0.x.zip. The error message: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}} ..

Error: Unexpected failure during upgrade. Check lcm-web.log or lcm.log. Details: PartnerUpgradeActionDetail(action=Failure occurred while running an upgrade for bundle: D1JM1_VxRail-8.0.200-Composite-Upgrade-Slim-Package-for-7.0.x.zip., message=Failure occurred while running an upgrade for bundle: D1JM1_VxRail-8.0.200-Composite-Upgrade-Slim-Package-for-7.0.x.zip. The error message: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}} .)ERROR:PartnerUpgradeActionDetail(action=Unexpected failure during upgrade. Check lcm-web.log or lcm.log., message=Upgrade failed. Unexpected failure during upgrade. Check lcm-web.log or lcm.log.)

VxRail plug-in error:
Failure occurred while running an upgrade for bundle: D1JM1_VxRail-8.0.200-Composite-Upgrade-Slim-Package-for-7.0.x.zip. The error message: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}} .
Upgrade failed. Unexpected failure during upgrade. Check lcm-web.log or lcm.log.
Upgrade preparation finished on ESXi host: <esxi fqdn>
Upgrade preparation is in progress on ESXi host:  <esxi fqdn>
Dell iSM for vSphere 7 upgrade is in progress on ESXi host:  <esxi fqdn>

lcm-web.log:
2024-03-11 19:24:46,859 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1418] Attempt 2/3 of vSAN access exception. but failed with error:com.vce.lcm.exception.LCMInternalException: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}}

2024-03-11 19:24:49,013 INFO [LCM] [lcm-node-0] c.d.v.l.d.p.i.MysticClusterReaderDO [MysticClusterReaderDO.java:200] type: STRETCHED CLUSTER
2024-03-11 19:24:49,041 INFO [LCM] [lcm-node-0] c.d.v.l.d.p.o.EventMicroWriter [EventMicroWriter.java:383] Successfully saved new event.event_id:588
2024-03-11 19:24:49,042 INFO [LCM] [lcm-node-0] c.d.v.l.d.p.o.EventMicroWriter [EventMicroWriter.java:93] Successfully saved event:NODE_UPGRADE_FAIL
2024-03-11 19:24:49,042 ERROR [LCM] [lcm-node-0] c.e.m.m.u.s.LcmNodeUpgradeServiceImpl [LcmNodeUpgradeServiceImpl.java:84] failed to upgrade node with exception:com.vce.lcm.exception.LCMException: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}}

        at com.vce.lcm.core.batch.BatchUpgrade.upgradeHost(BatchUpgrade.java:1793)
        at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1078)
        at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1400)
        at com.vce.lcm.core.upgrade.NodeUpgradeServiceImpl.performUpgrade(NodeUpgradeServiceImpl.java:102)
        at com.emc.mystic.manager.upgrade.executor.LcmNodeLegacyUpgradeExecutor.executeUpgrade(LcmNodeLegacyUpgradeExecutor.java:73)
        at com.emc.mystic.manager.upgrade.service.LcmNodeUpgradeServiceImpl$1.run(LcmNodeUpgradeServiceImpl.java:81)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vce.lcm.exception.LCMInternalException: {"VIB": {"Message": " [DependencyError] On platform embeddedEsx, VIB VMware_bootbank_nsx-esx-datapath_3.2.3.1.0-7.0.22104635 requires esx-base << 7.1, but the requirement cannot be satisfied within the ImageProfile. Please refer to the log file for more details.", "Success": false}}

Cause

NSX-T VIB upgrade, has failed on one of the nodes. The VIB version was reporting lower in the problematic node.
Run the command below on all the hosts to verify the VIB versions. 
esxcli software vib list | grep nsx

Resolution

To resolve this issue: 
  • Upgrade the NSX-T VIB from the NSX-T console using the "Resolve" button available against the NSX configuration install failed status for that node.
  • This pushes the NSX-T VIBs to the node and performs the installation. 
  • Confirm that the NSX-T VIBs are matching the cluster version in the ESXI host.
 #esxcli software vib list | grep nsx
  • Retry the upgrade from SDDC Manager once confirmed the correct version VIBs is installed.

Affected Products

VMWare Cloud on Dell EMC VxRail E560F
Article Properties
Article Number: 000223510
Article Type: Solution
Last Modified: 28 May 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.