VxRail: Upgrade Failed With Error "Schedule firmware using VxRail Platform Service failed with timeout"

Upgrade fails during firmware staging process with error: Schedule firmware using VxRail Platform Service failed with timeout

Lcm-web.log:

2022-03-15 14:57:04,936 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1223] Attempt 2/3 of vSAN access exception. but failed with error:
com.vce.lcm.exception.LCMInternalException: Schedule firmware using VxRail Platform Service failed with timeout.
 at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.checkStatus(EsxiFirmwareUpgrader.java:315)
        at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.scheduleBatchUpgradeStaging(EsxiFirmwareUpgrader.java:358)
        at com.vce.lcm.core.batch.BatchUpgrade.scheduleFirmwareStaging(BatchUpgrade.java:1809)
        .............................
        .............................
2022-03-15 14:57:05,152 ERROR [LCM] [lcm-node-0] c.e.m.m.u.s.LcmNodeUpgradeServiceImpl [LcmNodeUpgradeServiceImpl.java:83] failed to upgrade node with exception:
com.vce.lcm.exception.LCMException: Schedule firmware using VxRail Platform Service failed with timeout.
at com.vce.lcm.core.batch.BatchUpgrade.upgradeHost(BatchUpgrade.java:1573)
        at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1023)
        at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1205)
        .............................
        .............................​​​​​​​
Caused by: com.vce.lcm.exception.LCMInternalException: Schedule firmware using VxRail Platform Service failed with timeout.
        at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.checkStatus(EsxiFirmwareUpgrader.java:315)
        at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.scheduleBatchUpgradeStaging(EsxiFirmwareUpgrader.java:358)
        .............................
​​​​​​​         .............................
2022-03-15 14:57:57,331 ERROR [LCM] [lcm-core-0] c.v.l.c.u.ComponentUpgradeService [ComponentUpgradeService.java:310] Component VMWARE_ESXI upgrade failed due to Schedule firmware using VxRail Platform Service failed with timeout..
2022-03-15 14:57:57,399 ERROR [LCM] [lcm-core-0] c.v.l.a.LCMServiceImpl [LCMServiceImpl.java:1653] failed to upgrade: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, with exception:
com.vce.lcm.exception.LCMException: Schedule firmware using VxRail Platform Service failed with timeout.
        at com.vce.lcm.core.upgrade.ComponentUpgradeService.performComponentUpgrade(ComponentUpgradeService.java:329)
        at com.vce.lcm.api.LCMServiceImpl.performBatchUpgrade(LCMServiceImpl.java:1933)
        at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1285)
        .............................
       ​​​​​​​ .............................
2022-03-15 14:57:57,622 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:798] add error message status, profile: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, step: Upgrading, target: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, action: Failure occurred while running an upgrade for bundle: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip., detail messages: Failure occurred while running an upgrade for bundle: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip. The error message: Schedule firmware using VxRail Platform Service failed with timeout..

• Checking the iDRAC web UI, there were no firmware scheduled jobs during the staging process.
• Checked the platform_svc.log and found the following errors:

2022-03-15T12:09:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69
2022-03-15T12:10:14Z platform_svc: [Thread-2] INFO - Inventory update process is quiesced
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - start to check the upgrade progress ...
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - PTAgent response 200:
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO -     {
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO -         "inProgress": "true",
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO -         "progressPercentage": "69"
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO -     }
2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - start to check the upgrade progress ...
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - PTAgent response 200:
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO -     {
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO -         "inProgress": "true",
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO -         "progressPercentage": "69"
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO -     }
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Upgrade timeout. PTAGENT seems alive. We have to restart it.
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - no software vib change in progress, do not wait
2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Call ptagent start/restart command/etc/init.d/DellPTAgent restart
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - Error occurs during upgrading via PTAgent. Retrying ...
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -   Traceback (most recent call last):
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -     File "/opt/vxrail/bin/platforms/dell/plugin/__init__.py", line 1097, in do_pt_payload_fw_upgrade
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -       idrac_fw_packages)
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -     File "/opt/vxrail/bin/platforms/dell/plugin/__init__.py", line 1034, in _onetime_pt_payload_fw_upgrade
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -       raise Error('PTAgent firmware upgrade timeout')
2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR -   endpoints.error.Error: PTAgent firmware upgrade timeout

These errors in platform_svc.log were found looped retrying until the upgrade failed in the vCenter GUI.

Manually upgrading firmware through iDRAC Web UI, we can see one of the following errors:

Message: An internal error occurred. Unable to complete the specified operation.
Recommended Response Action: Retry the operation. If the issue persists, reset the iDRAC and retry the operation.

Message: The requested job cannot be scheduled because Lifecycle Controller is not enabled.
Recommended Response Action: Enable Lifecycle Controller and retry the operation. Lifecycle Controller can be enabled by using the F2 System Setup by pressing F2 during server startup, then select iDRAC Settings > Lifecycle Controlller and enable the Lifecycle Controller feature. Retry the operation after the server restarts.

The LifeCycle controller was found in Disabled status. With LCC disabled, iDRAC cannot upload and schedule the firmware to the job queue.

Run the command  racadm get LifeCycleController.LCAttributes  to verify:

racadm>>racadm get LifeCycleController.LCAttributes
#AutoDiscovery=Off
AutoUpdate=Disabled
BIOSRTDRequested=False
CollectSystemInventoryOnRestart=Enabled
#DiscoveryFactoryDefaults=Off
IgnoreCertWarning=On
IPAddress=
IPChangeNotifyPS=Off
#Licensed=No
LifecycleControllerState=Disabled
PartConfigurationUpdate=Apply Always
PartFirmwareUpdate=Match firmware of replaced part
ProvisioningServer=
#StorageHealthRollupStatus=2
#SystemID=0716
!!UserProxyPassword=******** (Write-Only)
UserProxyPort=80
UserProxyServer=
UserProxyType=HTTP
UserProxyUserName=

To resolve this issue: 

1. Connect to the iDRAC of the node that fails during firmware staging process through SSH.
2. Enable the LCC with command racadm set LifeCycleController.LCAttributes.LifecycleControllerState 1

racadm>>racadm set LifeCycleController.LCAttributes.LifecycleControllerState 1
Object value modified successfully

3. Validate the status of LCC again with command racadm get LifeCycleController.LCAttributes

racadm>>racadm get LifeCycleController.LCAttributes
#AutoDiscovery=Off
AutoUpdate=Disabled
BIOSRTDRequested=False
CollectSystemInventoryOnRestart=Enabled
#DiscoveryFactoryDefaults=Off
IgnoreCertWarning=On
IPAddress=
IPChangeNotifyPS=Off
#Licensed=No
LifecycleControllerState=Enabled <---- (correct value)
PartConfigurationUpdate=Apply Always 
PartFirmwareUpdate=Match firmware of replaced part
ProvisioningServer=
#StorageHealthRollupStatus=2
#SystemID=0716
!!UserProxyPassword=******** (Write-Only)
UserProxyPort=80
UserProxyServer=
UserProxyType=HTTP
UserProxyUserName=

4. Retry the upgrade and verify in the iDRAC UI that the firmware are being uploaded and scheduled in the job queue.

VxRail Appliance Family, VxRail Appliance Series, VxRail G Series Nodes, VxRail D Series Nodes, VxRail D560, VxRail D560F, VxRail E Series Nodes, VxRail E460, VxRail E560, VxRail E560F, VxRail E560N, VxRail E660, VxRail E660F, VxRail E660N , VxRail E665, VxRail E665F, VxRail E665N, VxRail G560, VxRail G560F, VxRail P Series Nodes, VxRail P470, VxRail P570, VxRail P570F, VxRail P580N, VxRail P670F, VxRail P670N, VxRail P675F, VxRail P675N, VxRail S Series Nodes, VxRail S470, VxRail S670, VxRail Software, VxRail V Series Nodes, VxRail V470, VxRail V570, VxRail V570F, VXRAIL V670F, VxRail VD-4510C, VxRail VD-4520C, VxRail VD Series Nodes, VxRail VE-660, VxRail VE-6615, VxRail VE-670, VxRail VP-760, VxRail VP-7625, VxRail VP-770, VxRail VS-760 ... View More about warranties View Less about warranties