PowerFlex 4.x RCM upgrade on HCI RG fails with Error: Failed to install VIBs

Summary: RCM upgrade fails during VMware VIB installation as the required files are no longer in the RCM/IC bundle.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

RCM upgrade fails. RCM upgrade fails during process_firmware_update:

Log in to thin-deployer pod and go to deployment.log of the depoyment_ID/Job_ID 
ERROR [2025-01-31T07:52:00.660421] 677128: service_deployment.rb:6081:in `process_firmware_update': Encountered an error during firmware update: RuntimeError: env --unset=RUBYOPT --unset=GEM_HOME --unset=RUBYLIB --unset=GEM_PATH --unset=BUNDLE_BIN_PATH RUBYLIB=/opt/service/lib:/opt/asm-deployer/lib:/opt/puppetlabs/puppet/lib:/opt/dependencies/dell-asm-util/lib PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/puppetlabs/puppet/bin:/opt/puppetlabs/bin puppet asm process_node --debug --trace --filename /opt/Dell/ASM/deployments/Job-2b551cc3-f546-4765-9eee-6416de02b254-0-0/resources/rackserver-cxjswf3.yaml --run_type apply --statedir /opt/Dell/ASM/deployments/Job-2b551cc3-f546-4765-9eee-6416de02b254-0-0/resources --always-override rackserver-cxjswf3 failed; output in /opt/Dell/ASM/deployments/Job-2b551cc3-f546-4765-9eee-6416de02b254-0-0/rackserver-abcdef3.out

ERROR [2025-01-31T07:56:00.831433] 677128: service_deployment.rb:7229:in `finalize_firmware_update': Failed to update the server!
INFO  [2025-01-31T07:56:00.831570] 677128: service_deployment.rb:7234:in `finalize_firmware_update': Firmware update status: Error
ERROR [2025-01-31T07:56:00.875669] 677128: service_deployment.rb:675:in `process': Firmware update failed for Job-2b551cc3-f546-4765-9eee-6416de02b254-0-0
ERROR [2025-01-31T07:56:00.875864] 677128: service_deployment.rb:676:in `process': ["/opt/asm-deployer/lib/asm/service_deployment.rb:7238:in `finalize_firmware_update'", "/opt/asm-deployer/lib/asm/service_deployment.rb:6095:in `process_firmware_update'", "/opt/asm-deployer/lib/asm/service_deployment.rb:526:in `process'", "/opt/asm-deployer/lib/asm.rb:385:in `block in process_deployment'", "org/jruby/RubyArray.java:2025:in `each_slice'", "/opt/asm-deployer/lib/asm.rb:366:in `block in process_deployment'", "/opt/dependencies/dell-asm-util/lib/asm/util.rb:22:in `block in execute_async'"]
INFO  [2025-01-31T07:56:00.876253] 677128: service_deployment.rb:678:in `process': Status: Error

From the .out file of the ESXi host, the installation failed while installing the following components:

/opt/Dell/ASM/deployments/<JOB_ID>/rackserver-<service_tag>.out 
** /opt/Dell/ASM/deployments/Job-2b551cc3-f546-4765-9eee-6416de02b254-0-0/rackserver-abcdef3.out **
Error: Failed to install VIB for
["https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/INT_bootbank_i40en_2.5.2.0-1OEM.700.1.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6WY7ee0a$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/DEL_bootbank_dcism_5.3.0.0.3289-DEL.700.0.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6RyLpejW$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/VMware_locker_tools-light_12.2.6.22229486-22348808.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6SW0V3E0$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/INT_bootbank_ixgben_1.15.1.0-1OEM.700.1.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6dkiwDeX$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/INT_bootbank_i40en_2.5.2.0-1OEM.700.1.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6WY7ee0a$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/DEL_bootbank_dcism_5.3.0.0.3289-DEL.700.0.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6RyLpejW$ [10[.]92[.]113[.]140]",
"https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/INT_bootbank_ixgben_1.15.1.0-1OEM.700.1.0.15843807.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6dkiwDeX$ [10[.]92[.]113[.]140]"]: Error RbVmomi::Fault:VimEsxCLICLIFault: EsxCLI.CLIFault.summary

Error: Fault error message: [" [VibFormatError]", " ('<http.client.HTTPResponse object at 0xe3d7cbf310>', \"Not a valid ar file (magic=b'')\")", "  
filename = https://10.92.113.140:443/httpshare/download/8aaa801394ba0c920194ba1b53b00004/vibs70/VMware_locker_tools-light_12.2.6.22229486-22348808.vib__;!!LpKI!glu2235RWNPqShl_xs0gxErxlqUao3Xy5zb_dK_dKz6ireUNS8hD7XuE_BEHR6rXQHNZg5cpfSE5oolHSF-bVBu-6SW0V3E0$ [10[.]92[.]113[.]140]", " Please refer to the log file for more details."]

Debug: Host Maintenance mode = true

Warning: Host is in maintenance mode. Exiting Maintenance mode...

Error: Failed to install VIBs
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'

Cause

The issue stems from the fact that the file VMware_locker_tools-light_12.2.6.22229486-22348808.VIB is 0 bytes in the RCM/IC file because DELL no longer provides these files. However, they are available in the VMware Patch depot but were skipped during the update process. 

Resolution

Any of the following workarounds may help.

  1. Uploading the VIB file using Single Component Upgrade (SCU):
    1. Extract VMware_locker_tools-light_12.2.6.22229486-22348808.VIB from the VMware patch depot.
    2. Enable Single Component Upgrade (SCU).
      Log in to any of the MVM nodes and run the following: 
      kubectl patch hr asmmanager -p '{"spec":{"values":{"appConfig": {"modify_compliance":"true"}}}}' -n powerflex --type=merge
    3. Go to PFMP UI → Compliance Version → Select the RCM bundle → Software Bundle → Select 'ESXi x.x Software Bundle' and click View → Select 'VMware Tools VIB for Windows and Linux guest' from the list of components and click Modify.
      Upload the VMware_locker_tools-light_12.2.6.22229486-22348808.VIB file and under 'New Bundle version' mention the following version 12.2.6.22229486-22348808. The version here would be the same as what you see under 'Bundle version'. Click Proceed:
      NOTE: You see this component with a 'Modified' tag.

       

    4. Perform the RCM/IC upgrade.
       
  2. Manually Install VMware Tools VIB:
    1. Manually install the VIB extracted from the offline update ZIP bundle downloaded from Broadcom: 
      # esxcli software vib install -v /<path_to_the_depot>/VMware_locker_tools-light_12.2.6.22229486-22348808.vib
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VMware_locker_tools-light_12.2.6.22229486-22348808
   VIBs Removed: VMware_locker_tools-light_12.3.5.22544099-22545121
   VIBs Skipped:
    2. Run the Inventory for the above ESXi host from the 'Resource' tab.
    3. Check the Compliance. The installed VIB is marked as compliant and skipped during the upgrade process.
    4. Perform the RCM/IC upgrade.
       
  3. Retry the RCM upgrade.  The second attempt should be successful.

Additional Information

Impacted Versions

PFMP 4.x

Fixed In Version

PFMP 4.6.1 

Affected Products

PowerFlex rack RCM Software
Article Properties
Article Number: 000283204
Article Type: Solution
Last Modified: 02 Jun 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.