VxRail: Upgrade Fails With Error "Failed to perform sub task: EnableVCVMSSHSubTask" or "Failed to perform sub task: UnlockCDROMSubTask"

Summary: Dell VxRail: Upgrade to 7.0.350 or above may fail at EnableVCVMSSHSubTask or UnlockCDROMSubTask when attempting to upgrade the VxRail Internal vCenter Server Appliance.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

VxRail upgrade to 7.0.350 or above may fail at EnableVCVMSSHSubTask or UnlockCDROMSubTask when attempting to upgrade the VxRail Internal vCenter Server Appliance.
Screenshot of update error in plugin 

short.term.log

2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[31m [ERROR] <68>[0m vsphere_connector_guestopt_impl.go ExecuteCommandOnVM() (131): Put ""https://169.254.x.x:443/guestFile?id=27&token=52dc49af-ed53-51d1-0898-6c21e791a33327"": dial tcp 169.254.x.x:443: i/o timeout"
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[36m [INFO] <68>[0m vm_repository_connector_impl.go ToggleSSH() (358): rc: -1"
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[36m [INFO] <68>[0m vm_repository_connector_impl.go ToggleSSH() (359): Stdout: "
2022-02-27-09:26:28 microservice.cmsvcupgrade-service ""
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[36m [INFO] <68>[0m vm_repository_connector_impl.go ToggleSSH() (360): Stderr: "
2022-02-27-09:26:28 microservice.cmsvcupgrade-service ""
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[31m [ERROR] <68>[0m vm.go ToggleSSH() (364): Put ""https://169.254.x.x:443/guestFile?id=27&token=52dc49af-ed53-51d1-0898-6c21e791a33327"": dial tcp 169.254.x.x:443: i/o timeout"
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[31m [ERROR] <68>[0m enable_vc_vm_ssh_subtask.go PerformSubTask() (80): Put ""https://169.254.x.x:443/guestFile?id=27&token=52dc49af-ed53-51d1-0898-6c21e791a33327"": dial tcp 169.254.x.x:443: i/o timeout"
2022-02-27-09:26:28 microservice.cmsvcupgrade-service "2022-02-27 09:26:28[31m [ERROR] <68>[0m upgrade_flow.go PerformSubTasks() (215): Fail to run sub task: [EnableVCVMSSHSubTask], error: [Put ""https://169.254.x.x:443/guestFile?id=27&token=52dc49af-ed53-51d1-0898-6c21e791a33327"": dial tcp 169.254.x.x:443: i/o timeout]"


Or

"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.657905947Z stderr F 2022-07-29 07:45:16[31m [ERROR] <75>[0m vsphere_connector_guestopt_impl.go ExecuteCommandOnVM() (131): Put ""https://169.254.x.x:443/guestFile?id=15&token=524a1709-f64a-bfae-2b76-7f8a144f09a015"": dial tcp 69.254.xxx:443: connect: connection refused"
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.75273638Z stderr F 2022-07-29 07:45:16[36m [INFO] <75>[0m vm_repository_connector_impl.go ToggleSSH() (358): rc: -1"
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752786866Z stderr F 2022-07-29 07:45:16[36m [INFO] <75>[0m vm_repository_connector_impl.go ToggleSSH() (359): Stdout: "
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752790794Z stderr F "
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752795238Z stderr F 2022-07-29 07:45:16[36m [INFO] <75>[0m vm_repository_connector_impl.go ToggleSSH() (360): Stderr: "
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752798003Z stderr F "
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752839333Z stderr F 2022-07-29 07:45:16[31m [ERROR] <75>[0m vm.go ToggleSSH() (364): Put ""https://169.254.x.x:443/guestFile?id=15&token=524a1709-f64a-bfae-2b76-7f8a144f09a015"": dial tcp 169.254.69.254.xxx:443: connect: connection refused"
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752890818Z stderr F 2022-07-29 07:45:16[31m [ERROR] <75>[0m enable_vc_vm_ssh_subtask.go PerformSubTask() (80): Put ""https://69.254.xxx:443/guestFile?id=15&token=524a1709-f64a-bfae-2b76-7f8a144f09a015"": dial tcp 169.254.57.111:443: connect: connection refused"
"2022-07-29 07:45:17,242" microservice.cmsvcupgrade-service "2022-07-29T07:45:16.752937517Z stderr F 2022-07-29 07:45:16[31m [ERROR] <75>[0m upgrade_flow.go PerformSubTasks() (215): Fail to run sub task: [EnableVCVMSSHSubTask], error: [Put ""https://69.254.xxx:443/guestFile?id=15&token=524a1709-f64a-bfae-2b76-7f8a144f09a015"": dial tcp 169.254.xxx:443: connect: connection refused]"

Cause

ESXi hosts have IPv4 address configured on vmk0, and the address is not accessible from VxRail Manager, like 169.254.x.x.
Usually, these hosts have been upgraded from 4.5 at some point in their lifetime. In 4.7.x VxRail IPv4 Management was moved to vmk2. As a result, it is possible for a vmk0 to have a left-over setting which still tags vmk0 as IPv4 management address, which as a result will set an 169.254.x.x address on vmk0

Resolution

  1. Disable IPv4 for vmk0 on all ESXi hosts, vmk0 should have IPv6 address only.
  2. Retry upgrade

Affected Products

VxRail, VxRail Appliance Series, VxRail Software
Article Properties
Article Number: 000201456
Article Type: Solution
Last Modified: 11 Nov 2025
Version:  7
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.