VxRail: Failed to Upgrade vCenter Server With Error: Failed to perform sub task: EnableVCVMSSHSubTask, see logs for detail errors

VxRail upgrade fails with error:
VxRail Update ran into a problem....
Failed to upgrade vCenter Server: Failed to perform sub task: EnableVCVMSSHSubTask, see logs for detail errors..
Upgrade failed. Unexpected failure during upgrade, check lcm-web.log or lcm.log


The error occurs before vCenter .ova gets deployed. 

 

lcm-web.log:

2022-04-11 14:41:25,167 ERROR [LCM] [lcm-vc-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:82] Failed to execute task VCLCMUpgradeTask.
java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMInternalException: Failed to upgrade vCenter Server: Failed to perform sub task: EnableVCVMSSHSubTask, see logs for detail errors..
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:72)
............
	at com.vce.lcm.core.upgrade.vc.AbstractVCUpgrader.performUpgrade(AbstractVCUpgrader.java:107)
	at com.vce.lcm.core.upgrade.ComponentUpgradeService.performComponentUpgrade(ComponentUpgradeService.java:199)
	at com.vce.lcm.core.upgrade.VcUpgradeServiceImpl.performUpgrade(VcUpgradeServiceImpl.java:108)
	at com.emc.mystic.manager.upgrade.service.LcmVcUpgradeServiceImpl$1.run(LcmVcUpgradeServiceImpl.java:86)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vce.lcm.exception.LCMInternalException: Failed to upgrade vCenter Server: Failed to perform sub task: EnableVCVMSSHSubTask, see logs for detail errors..
	at com.vce.lcm.core.upgrade.vc.task.VCLCMUpgradeTask.WaitForVCLCMUpgradeComplete(VCLCMUpgradeTask.java:330)
	at com.vce.lcm.core.upgrade.vc.task.VCLCMUpgradeTask.perform(VCLCMUpgradeTask.java:118)
	at com.vce.lcm.task.UpgradeTask.execute(UpgradeTask.java:58)
	at com.vce.lcm.task.SimpleUpgradeTaskExecutor.lambda$execute$0(SimpleUpgradeTaskExecutor.java:65)
	...

The Link Aggregation Control Protocol (LACP) Support level on a vSphere Distributed Switch is set to basic. 



 

Check KB VxRail: Upgrade Fails With Error "Failed to perform sub task: EnableVCVMSSHSubTask" or "Failed to perform sub task: UnlockCDROMSubTask" first, if it does not help, then apply below workaround.
 

1. Take a snapshot of the VxRail Manager (VxRM) virtual machine.
2. Connect VxRM using CLI navigator.
3. Go to the directory /var/lib/vmware-marvin/
4. Locate the file named lcm_advanced_mode.properties
5. Modify the file content and change the value from true to false for lcmProperties.vclcm.upgrade.enabled 
6. Change the value from false to true for lcmProperties.upgrade.advanced.mode.on

   cd /var/lib/vmware-marvin
   vi lcm_advanced_mode.properties 
   lcmProperties.skip.witness.upgrade.in.vlcm=false 
   feature.upgrade.on.l3.network=on 
   feature.compliance.daily.check=on 
   lcmProperties.upgrade.advanced.mode.on=true 
   lcmProperties.vclcm.upgrade.enabled=false 
   lcmProperties.upgrade.skip.adc.advanced.mode.on=false 
   lcmProperties.force.legacy.batch.upgrade=false

7. Save the changes.
8. Change the vDS LACP mode to "Enhanced Mode" following VMware KB: https://kb.vmware.com/s/article/2051311
9. Restart VxRM services

   systemctl restart vmware-marvin 
   systemctl restart runjars

10. Retry the upgrade.
11. After completing upgrade, revert modified values in file /var/lib/vmware-marvin/lcm_advanced_mode.properties

    lcmProperties.upgrade.advanced.mode.on=false
    lcmProperties.vclcm.upgrade.enabled=true​​

1. VXEE-9804
2. VXEE-10310