Dell EMC VCF on VxRail: How to Install NSX VIB Manually if the Installation Fails on ESXi during the VCF Upgrade.

Summary: This article explains how to Install NSX VIB Manually if the Installation Fails on ESXi during the VCF Upgrade.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Instructions

First of all, it is advised to follow this article: https://kb.vmware.com/s/article/81203

If the above article does not fix the Issue, proceed with the steps documented below.

VIB: A VIB is an ESXi software package. VMware and its partners package solutions, drivers, CIM providers, and applications that extend the ESXi platform as VIBs. VIBs are available in software depots.
https://blogs.vmware.com/vsphere/2011/09/whats-in-a-vib.html

Image Profile: An image profile defines an ESXi image and consists of VIBs. An image profile always includes a base VIB, and might include more VIBs.

Software Depot: A software depot is a collection of VIBs and image profiles. The software depot is a hierarchy of files and folders and can be available through an HTTP URL (online depot) or a ZIP file (offline depot).

A VIB has three parts:
VIB: vSphere Installation BundleFile Archive - the main file. The file which gets deployed to the ESXi host.
XML descriptor file: Has important info about requirements for installing the VIB. (dependencies, compatibility, reboot necessary? )
Signature File: Signature which verifies the level of trust (Integrity, Information about the creator and verification that has been done).
 
• Installation of NSX VIBs that enable the Distributed Firewall and VXLAN are essential for a working NSX environment. vSphere ESXi Agent Manager deploys vSphere installation bundles (VIBs) onto ESXi hosts.
• Prior to NSX 6.2.0, there was a single URL on NSX Manager from which VIBs for a certain version of the ESXi Host could be found. (Meaning the administrator only needed to know a single URL, regardless of NSX version. In NSX 6.2.0 and later, the new NSX VIBs are available at different URLs.
 
Below are the ESXi hosts that were failing with the installation of NSX VIBS on the ESXi hosts.

Navigate to Network and Security →Installation and Upgrade → Host Preparation tab from the Menu in vSphere Web Client.
Host with Not ready error
The Status of "Not Ready" means that the NSX VIB did not get Installed.

To manually install NSX 6.3.0 VIBS on the ESXi 6.5 hosts. We must identify the path of the NSX VIBS to get it downloaded and installed on the ESXi host. You can identify the path of the VIBS using the below URL.

https://<NSX-Manager-IP>/bin/vdn/nwfabric.properties

Note: VIBs are hosted on NSX Manager and are also available as a .zip file. The downloadable .zip file differs based on NSX and ESXi version
# 6.0 VDN EAM Info
VDN_VIB_PATH.1=/bin/vdn/vibs-6.4.0/6.0-7263674/vxlan.zip
VDN_VIB_VERSION.1=7263674
VDN_HOST_PRODUCT_LINE.1=embeddedEsx
VDN_HOST_VERSION.1=6.0.*
# 6.5 VDN EAM Info
VDN_VIB_PATH.2=/bin/vdn/vibs-6.4.0/6.5-7263674/vxlan.zip
VDN_VIB_VERSION.2=7263674
VDN_HOST_PRODUCT_LINE.2=embeddedEsx
VDN_HOST_VERSION.2=6.5.*
# 6.6 VDN EAM Info
VDN_VIB_PATH.3=/bin/vdn/vibs-6.4.0/6.6-7263674/vxlan.zip
VDN_VIB_VERSION.3=7263674
VDN_HOST_PRODUCT_LINE.3=embeddedEsx
VDN_HOST_VERSION.3=6.6.*
# 6.6.2 VDN EAM Info for SDDC
VDN_VIB_PATH.4=/bin/vdn/vibs-6.4.0-sddc/6.6-7263674/vxlan.zip
VDN_VIB_VERSION.4=7263674
VDN_HOST_PRODUCT_LINE.4=embeddedEsx.sddc
VDN_HOST_VERSION.4=6.6.2
# Single Version associated with all the VIBs pointed by above VDN_VIB_PATH(s)
VDN_VIB_VERSION=6.4.0.7269135
# Legacy vib location. Used by code to discover avaialble legacy vibs.
LEGACY_VDN_VIB_PATH_FS=/common/em/components/vdn/vibs/legacy/

Note: The VIBs installed on a host depend on the NSX and ESXi versions:
Table with VIBs installed, NSX and ESXi versions

NSX Build Mappings:

NSX Build Mappings on bin/vdn/nwfabric.properties

As we are going to manually install NSX VIBS copy the VIB path for ESXi hosts and Type the URL in the browser. You need the access the VIB path based on your ESXi version.


https://<NSX-Manager-IP>/bin/vdn/vibs-6.3.0/6.5-4987428/vxlan.zip [This URL changes based on different ESXi Versions]

Save the vxlan.zip file on your local system. Extract the vxlan.Zip file using WinZip. Copy the VIB files into the ESXi host using WinSCP. I have copied the VIB files under the /tmp directory of the ESXi hosts.

Or you can get the NSX vip from the VxRail upgrade bundle.

Upgrade bindle on a Windows environment

Place the Host into Maintenance mode before Installing the VIB. To enter maintenance mode through CLI run below cmd.
vim-cmd /hostsvc/maintenance_mode_enter
To Exit out of maintenance mode, run below cmd
vimsh -n -e /hostsvc/maintenance_mode_exit
Alternatively, you can also run below cmd for maintenance mode.
esxcli system maintenanceMode set –enable true
esxcli system maintenanceMode set –enable false
To check if the Host is in Maintenance Mode, run below command on the ESXi Host by logging in as a root user.
vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode OR vimsh -n -e /hostsvc/hostsummary | grep inMaintenanceMode
Find out if there are no VMs running on the ESXi host using below cmd.
esxcli vm process list
Install the “esx-vxlan” VIB on the ESXi host using the below command:
esxcli software vib install -v /tmp/VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428.vib
 OR
esxcli software vib install –no-sig-check -v /tmp/VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648.vib
If you receive any nsx-api related errors during the installation, you can add “–force” parameter with the above command
esxcli software vib install -v /tmp/VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428.vib --force

How to Verify NSX VIB Installation on ESXi Host
• NSX Manager is responsible for deploying the NSX Controller cluster, ESXi hosts preparation by pushing vSphere Installation Bundles (VIBs) to enable VXLAN, distributed routing, distributed firewall, and a user world agent used to communicate at the control-plane level. The functionality of the user world agent is highly critical and any failures will have a direct impact on the control plane learning, which eventually affects data plane traffic.
• The user world agent (UWA) is composed of the netcpad and vsfwd daemons on the ESXi host. UWA Uses SSL to communicate with the NSX Controller on the control plane. UWA Mediates between NSX Controller and the hypervisor kernel modules, except the distributed firewall. Communication related to NSX between the NSX Manager instance or the NSX Controller instances and the ESXi host happen through the UWA. UWA Retrieves information from NSX Manager through the message bus agent.
 
Netcpa:

It is a user world agent that communicates with the NSX control plane, and the netcpa service should be up and running on the NSX-prepared ESXi host. If the functionality is impacted, we will certainly experience routing and switching issues in the NSX environment, and the ESXi host will not learn new routes from the time the netcpa service was down.

We can verify the status of User World agents (UWA) from CLI.
/etc/init.d/netcpad status

User World Agents (UWA) maintain the logs at /var/log/netcpa.log
/var/log/netcpa.log

In the /var/log/netcpa.log file on the ESXi host, we can see the complete netcpa logs.

netcpa logs

Check if the netcpa configuration file is showing all the controllers. Use the following command to check controller details in the configuration file:
cat /etc/vmare/netcpa/config-by-vsm.xml

The following screenshot depicts the config file output, with controller IP and SSL certificate thumbprint information:

config file output

Anytime we are facing issues with the netcpa service, restart the service to confirm if that fixes the issue. To restart the netcpa service, we must complete the following steps in order:
1. Log in as root to the ESXi host through SSH or through the DCUI console.
2. Run the /etc/init.d/netcpad restart command to restart the netcpa agent on the ESXi host.

Vsfwd:
NSX distributed firewall is a hypervisor-integrated firewall and apart from the point that the host should have a firewall VIB  installed, there should be a vsfwd daemon process up and running for proper message bus communication with NSX Manager.

The following command and screenshot shows a stateful firewall status on the ESXi host:
 
/etc/init.d/vShield-Stateful-Firewall status
Putty image

To check the active message bus session with NSX Manager the following command and screenshot depicts an active session with NSX Manager (172.16.1.5):
esxcli network ip connection list | grep 5671
Putty result

A potential failure can happen if port 5671 is not opened between the ESXi host and NSX Manager.

All logs related to vsfwd will be at the following location on the ESXi host.
/var/log/vsfwd.log
To verify NSX VIB Installation, run below cmd.
[root@ppevh309:~] esxcli software vib get --vibname esx-nsxv
VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648
Name: esx-nsxv
Version: 6.7.0-0.0.19279648
Type: bootbank
Vendor: VMware
Acceptance Level: VMwareCertified
Summary: NSX datapath and host tools
Description: This package loads module and configures firewall for NSX-v logical networking.
ReferenceURLs:
Creation Date: 2022-01-28
Depends: esx-base << 7.1, nsx-api = 2.1, vmkapi_2_5_0_0
Conflicts:
Replaces: esx-traceflow, esx-dvfilter-switch-security, esx-bfd, esx-vdpi, esx-vxlan, esx-vsip
Provides: com.vmware.vxlan = 1.0.0.0-nsx, com.vmware.switchsecurity = 1.0.0.0, com.vmware.traceflow = 1.0.0.0, com.vmware.bfd = 1.0.0.0, vsip = 1.0.0-0
Maintenance Mode Required: True
Hardware Platforms Required:
Live Install Allowed: True
Live Remove Allowed: True
Stateless Ready: True
Overlay: False
Tags:
Payloads: esx-nsxv
Log in to ESXi Shell or SSH as root and run below command to check the newly installed VIB.
esxcli software vib list
Name Version Vendor Acceptance Level Install Date
----------------- --------------------------- ------ ---------------- ------------
ata-pata-amd 0.3.10-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-04
ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-04
ata-pata-cmd64x 0.2.5-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-04
ata-pata-hpt3x2n 0.3.4-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-04
The Log messages for VIB Installation can be found at /var/log/esxupdate.log on the ESXi Host.
[root@ESXi01:/var/log] less esxupdate.log | grep "nsxv"
2022-05-24T07:42:44Z esxupdate: Mware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-2.60.13981272, VMW_bootbank_nhpsa_2.0.22-3vmw.670.2.60. 13981272, VMware_bootbank_vmware-fdm_6.7.0-19300125, VMW_bootbank_sata-ahci_3.0-26vmw.670.2.60.13981272, VMW_bootbank_net-mlx4-core_1.9.7. 0-1vmw.670.2.60.13981272, VMware_bootbank_qlnativefc_3.1.8.0-5vmw.670.3.73.14320388, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3.89 .15160138, VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMW_bootbank_pvscsi_0.1-2vmw.670.2.60.13981272, VMW_bootbank_net-e1000_8.0.3.1-5vm w.670.3.112.16701467, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.2.60.13981272, BCM_bootbank_bnxtroce_216.0.65.0-1OEM.670.0.0. 8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.2.60.13981272, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.2.60.13981272, VMware_
bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.60.13981272, VMW_bootbank_xhci-xhci_1.0-3vmw.670.2.60.13981272,
2022-05-24T07:43:01Z esxupdate: 16319420: BootBankInstaller.pyc: DEBUG: About to write payload 'esx-nsxv' of VIB VMware_bootbank_esx-nsxv_ 6.7.0-0.0.17493714 to '/tmp/stagebootbank'
2022-05-24T07:43:01Z esxupdate: 16319420: BootBankInstaller.pyc: DEBUG: Using /bootbank/esx_nsxv.v00 from bootbank as source
2022-05-24T07:43:52Z esxupdate: 16321124: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-17493714.xml
2022-05-24T07:43:52Z esxupdate: 16321124: Metadata.pyc: INFO: Processing vib xml esx-nsxv-6656408272306348672.xml
2022-05-24T07:43:54Z esxupdate: 16321124: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-17493714.xml
2022-05-24T07:43:54Z esxupdate: 16321124: Metadata.pyc: INFO: Processing vib xml esx-nsxv-6656408272306348672.xml
2022-05-24T07:43:54Z esxupdate: 16321124: imageprofile: DEBUG: VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714 is being removed from Image Profile (Updated) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard -->> ( This is where older VIB gets removed )
2022-05-24T09:44:07Z esxupdate: 16358579: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-17493714.xml
2022-05-24T09:44:07Z esxupdate: 16358579: Metadata.pyc: INFO: Processing vib xml esx-nsxv-6656408272306348672.xml
2022-05-24T09:44:08Z esxupdate: 16358579: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-17493714.xml
2022-05-24T09:44:08Z esxupdate: 16358579: Metadata.pyc: INFO: Processing vib xml esx-nsxv-6656408272306348672.xml
2022-05-24T09:44:08Z esxupdate: 16358579: imageprofile: DEBUG: VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714 is being removed from Image Profile (Updated) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard
2022-05-25T04:50:56Z esxupdate: 16722224: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-19279648.xml
2022-05-25T04:50:56Z esxupdate: 16722224: Metadata.pyc: INFO: Processing vib xml esx-nsxv-4024525266687084002.xml
2022-05-25T04:50:57Z esxupdate: 16722224: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-19279648.xml
2022-05-25T04:50:57Z esxupdate: 16722224: Metadata.pyc: INFO: Processing vib xml esx-nsxv-4024525266687084002.xml
2022-05-25T04:50:57Z esxupdate: 16722224: imageprofile: INFO: Adding VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648 to ImageProfile (Upda ted) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard -->> ( This is where new VIB gets Installed )
2022-05-25T04:50:57Z esxupdate: 16722224: imageprofile: DEBUG: VIBs {'VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714'} are replaced by VIB VM ware_bootbank_esx-nsxv_6.7.0-0.0.19279648, removing them from ImageProfile (Updated) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standa rd
2022-05-25T04:50:57Z esxupdate: 16722224: imageprofile: DEBUG: VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714 is being removed from Image Profile (Updated) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard
2022-05-25T11:02:05Z esxupdate: 16848444: LiveImageInstaller: DEBUG: Trying to unmount payload [esx-nsxv] of VIB VMware_bootbank_esx-nsxv_ 6.7.0-0.0.17493714
2022-05-25T11:02:05Z esxupdate: 16848444: LiveImageInstaller: DEBUG: Copying tardisk from /tardisks/esx_nsxv.v00 to /tmp/tardiskbackup/esx _nsxv.v00
2022-05-25T11:02:05Z esxupdate: 16848444: LiveImageInstaller: DEBUG: Unmounting esx_nsxv.v00...
2022-05-25T11:02:05Z esxupdate: 16848444: vmware.runcommand: INFO: runcommand called with: args = 'rm /tardisks/esx_nsxv.v00', outfile = ' None', returnoutput = 'True', timeout = '0.0'.
2022-05-25T11:02:21Z esxupdate: 16851533: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-19279648.xml
2022-05-25T11:02:21Z esxupdate: 16851533: Metadata.pyc: INFO: Processing vib xml esx-nsxv-4024525266687084002.xml
2022-05-25T11:02:23Z esxupdate: 16851533: Metadata.pyc: INFO: Processing bulletin xml nsxv-datapath-6.7-19279648.xml
2022-05-25T11:02:23Z esxupdate: 16851533: Metadata.pyc: INFO: Processing vib xml esx-nsxv-4024525266687084002.xml
2022-05-25T11:02:23Z esxupdate: 16851533: imageprofile: INFO: Adding VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648 to ImageProfile (Upda ted) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard
2022-05-25T11:02:25Z esxupdate: 16851544: Transaction: INFO: Final list of VIBs being installed: VMware_bootbank_esx-nsxv_6.7.0-0.0.192796 48
2022-05-25T11:02:25Z esxupdate: 16851544: imageprofile: INFO: Adding VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648 to ImageProfile (Upda ted) ESXi-6.7.0-20210304001-67p05_17702396-d4a8f2b5-standard
2022-05-25T11:02:25Z esxupdate: VMW_bootbank_sata-sata-sil_2.3-4vmw.670.2.60.13981272, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3. 89.15160138, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.2.60.13981272, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-3vmw.670.2.60.13981272, VMwar e_bootbank_esx-nsxv_6.7.0-0.0.19279648, VMW_bootbank_elxnet_11.4.1097.0-5vmw.670.3.73.14320388, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0 .0-7vmw.670.2.60.13981272, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.2.60.13981272, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.2.60.13 981272, VMW_bootbank_vmkata_0.1-1vmw.670.2.60.13981272, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.2.60.13981272, VMW_bootbank_qfle3i_ 1.0.2.3.9-3vmw.670.2.60.13981272, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.60.13981272, VMW_bootbank_brcmfcoe_11.4.1078.25-14vmw.670.3.7 3.14320388, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.2.60.13981272, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.2.60
2022-05-25T11:02:25Z esxupdate: 16851544: HostImage: DEBUG: --- Stage: LiveImageInstaller adding [VMware_bootbank_esx-nsxv_6.7.0-0.0.1927 9648], removing []
2022-05-25T11:02:26Z esxupdate: 16851544: HostImage: INFO: Attempting to download VIB esx-nsxv
2022-05-25T11:02:26Z esxupdate: 16851544: downloader: INFO: Downloading https://ppaim113.protected.internal:443/eam/vib?id=25c2ee91-8e25-4 373-8c07-a1ea83ae76b9 to /tmp/vibdownload/VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648.vib
2022-05-25T11:02:26Z esxupdate: 16851544: root: DEBUG: Verifying VIB VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648 signature #2
2022-05-25T11:02:26Z esxupdate: 16851544: LiveImageInstaller: INFO: Firewall Refresh Trigger is required to install/remove VMware_bootbank _esx-nsxv_6.7.0-0.0.19279648
2022-05-25T11:02:27Z esxupdate: 16851544: LiveImageInstaller: DEBUG: Starting to enable VIBs: VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648
2022-05-25T11:02:27Z esxupdate: 16851544: LiveImageInstaller: DEBUG: Live installing esx-nsxv-6.7.0-0.0.19279648
2022-05-25T11:02:27Z esxupdate: 16851544: LiveImageInstaller: DEBUG: Trying to mount payload [esx-nsxv]
2022-05-25T11:02:27Z esxupdate: 16851544: LiveImageInstaller: DEBUG: Mounting esx_nsxv.v00...
2022-05-25T11:02:27Z esxupdate: 16851544: vmware.runcommand: INFO: runcommand called with: args = '['/usr/lib/vmware/secureboot/bin/secure Mount.py', 'VMware_bootbank_esx-nsxv_6.7.0-0.0.19279648', 'esx-nsxv', '/tmp/stageliveimage/data/esx_nsxv.v00']', outfile = 'None', returno utput = 'True', timeout = '0.0'.

Additional Information

More articles to follow up on this topic:

“esxcli software VIB ” commands to patch an ESXi 6.x/7.x host
https://kb.vmware.com/s/article/2008939

VIB's Troubleshooting
https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.troubleshooting.doc/GUID-C940AD16-84AA-4AC6-AEB5-88DF448FED1B.html

Affected Products

VxRail Software
Article Properties
Article Number: 000218389
Article Type: How To
Last Modified: 13 Feb 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.