VxRail: LCM van VxRail 7.0.520 of 7.0.521 geeft foutmelding: time-out voor reparatie van vSAN-schijfobject ophalen

Het tabblad VxRail LCM-gebruikersinterface van vCenter plug-inupdates toont: 

Error message: 
VxRail Update ran into a problem...
Failed to retrieve the existing value of the vSAN disk object repair timeout.

In de VxRail Manager wordt het /var/log/mystic/lcm-web.log Toont:

2024-09-27T15:19:21.429Z <7ff91b9f32f61676bfb7758d9af41713> lcm [ERROR] <94> ComponentUpgradeService.java performComponentUpgrade() (323): Component VMWARE_ESXI upgrade failed due to Failed to retrieve the existing value of the vSAN disk object repair timeout..
2024-09-27T15:19:21.429Z <7ff91b9f32f61676bfb7758d9af41713> lcm [INFO] <94> LCMServiceImpl.java handleUpgradeException() (1812): handling upgrade exception.
2024-09-27T15:19:21.429Z <7ff91b9f32f61676bfb7758d9af41713> lcm [INFO] <94> LCMServiceImpl.java handleUpgradeException() (1817): Error from vc or node upgrade, sync the composite file
...
2024-09-27T15:19:21.467Z <7ff91b9f32f61676bfb7758d9af41713> lcm [ERROR] <94> LCMServiceImpl.java handleUpgradeException() (1855): failed to upgrade: VXRAIL_COMPOSITE-7.0.531-28777448_for_7.0.x.zip,
 with exception:
com.vce.lcm.exception.LCMInternalException: Failed to retrieve the existing value of the vSAN disk object repair timeout.
        at com.vce.lcm.core.upgrade.ComponentUpgradeService.performComponentUpgrade(ComponentUpgradeService.java:336)
        at com.vce.lcm.api.LCMServiceImpl.performBatchUpgrade(LCMServiceImpl.java:2152)
...
024-09-27T15:19:21.514Z <7ff91b9f32f61676bfb7758d9af41713> lcm [INFO] <94> VirtualApplianceUpgradeParameterStatusHandler.java notifyError() (102): Notify errors for upgrade bundle UpgradeBundleProfile{name='VXRAIL_COMPOSITE-7.0.531-28777448_for_7.0.x.zip', description='null', sizeInKBytes='15664544', bundleFileName='VXRAIL_COMPOSITE-7.0.531-28777448_for_7.0.x.zip', version='7.0.531', build='28777448', targetHardwareInfo='', targetModelInfo='com.vce.commons.vcm.state.CompositeManifestFile$TargetModelInfo@3901ec9', replaceTargetInfo='com.vce.commons.vcm.state.CompositeManifestFile$ReplaceTargetInfo@16a5c26c', targetNicModelInfo='com.vce.commons.vcm.state.CompositeManifestFile$TargetNicModelInfo@2949e0b9', removeUpgradeVIB='false', forceInstall='false', skipList='[]', upgradeTime='0', rebootFlag='false', isUpgradeComponent='false', totalEstimated='100668', totalUsed='54511', minimumVxRailVersion='7.0.010', patchType=''}, upgradeStep UPGRADE, upgradeTarget UpgradeTarget{ComponentType=VXRAIL_COMPOSITE, name='esxxxx.domain.country', isPipelineTask='false'}, action Retrieve the value of the vSAN disk object repair timeout., message Failed to retrieve the existing value of the vSAN disk object repair timeout.

Extra foutdetails helpen bij het identificeren van de getroffen host en het extra symptoom Te veel openstaande bewerkingen:

2024-09-27T15:18:29.011Z <7ff91b9f32f61676bfb7758d9af41713> lcm [INFO] <374> BatchUpgrade.java updateVsanClomTimeoutValueBeforeUpgrade() (881): Obtaining the existing vSAN Clom timeout value for th
e host : esxxxx.domain.country
...
2024-09-27T15:18:33.070Z <7ff91b9f32f61676bfb7758d9af41713> lcm [WARN] <374> ScriptServiceImpl.java runLocalScript() (507): [/etc/vmware-marvin/scripts/lcm/scripts/update-vsan-clom-repair-timeout.p
y@localhost]: <stderr> Traceback (most recent call last):
  File "/usr/lib/vmware-marvin/marvind/temp/3741351928508979296.tmp", line 154, in <module>
    main()
  File "/usr/lib/vmware-marvin/marvind/temp/3741351928508979296.tmp", line 149, in main
    get_timeout_value(options)
  File "/usr/lib/vmware-marvin/marvind/temp/3741351928508979296.tmp", line 133, in get_timeout_value
    result = advanced_sys_settings.List(False, VSAN_PROPERTY)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pyVmomi/ManagedMethodExecutorHelper.py", line 74, in InvokeMethod
    result = self.mme.ExecuteSoap(moid=moid,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pyVmomi/SoapAdapter.py", line 1397, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
    ^^^^^^^^^
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'A general system error occurred: Too many outstanding operations',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'Too many outstanding operations'
}

Een codeprobleem in VxRail Release 7.0.520 en 7.0.521 veroorzaakt esxcli host-opdrachten om na verloop van tijd vast te lopen. De query in de logboeken retourneert:

reason = 'Too many outstanding operations'

Dit resulteert in een LCM-fout, het upgradeproces kan geen VSAN-configuratiewaarde ophalen via esxcli.
 

VxRail release 7.0.531 biedt een permanente oplossing voor dit probleem. Releaseopmerkingen zijn te vinden op: Dell VxRail 7.0.x Release Notes | Dell VS (aanmelding bij klant vereist)

De gebruiker moet het hostd management service op het defecte knooppunt voordat u het VxRail LCM-upgradescript opnieuw start. Hiermee worden de openstaande aanvragen gewist en kan de upgrade worden voortgezet.

/etc/init.d/hostd restart

Voor begeleiding bij het opnieuw opstarten van de ESXi hostd service, referentie Broadcom-artikel De beheeragenten opnieuw opstarten in ESXi. 

Als hulp nodig is bij het bekijken van de logboeken, het identificeren van de getroffen host of het opnieuw starten van de service, neem dan contact op met Dell Support.

Als STIG is toegepast, lees dan het artikel VxRail: Onjuiste bestandsmachtigingen voor sommige bibliotheken Tijdens een upgrade naar 7.0.400 met STIG, als mogelijke oorzaak van deze fout