VxRail: Unable To Upgrade On-Disk Format Version

The following symptoms have been seen:

• Unable to upgrade vSAN On-Disk format version.
• vSAN on disk remains with older version after vSAN cluster upgrade to 8.0.x.
• Upgrade fails with generic message.
• There are warnings observed on Skyline health.
• Upgrade pre-checks passed without any issues, but upgrade still fails from UI and CLI.

Logs and error messages include:
 

2025-04-25 11:14:21 +0000: Running precondition checks ...
2025-04-25 11:14:21 +0000: Passed precondition checks
2025-04-25 11:14:21 +0000: 
2025-04-25 11:14:21 +0000: Target file system version: v20
2025-04-25 11:14:21 +0000: Disk mapping decommission mode specified: evacuateAllData
2025-04-25 11:14:24 +0000: Upgrade tool stopped due to error, please address reported issue and re-run the tool again to finish upgrade.

2025-05-14T06:23:52.737Z ERROR vsan-mgmt[1402576] [VsanVcDiskFormatConverterImpl::_Run opID=agw-0115827-5ae9] Disk conversion thread met unexpected error

2025-05-14T06:23:52.736Z error vpxd[3650220] [Originator@6876 sub=Default opID=agw-0115827-5ae9-de] [VpxLRO] -- ERROR lro-67767942 -- 52659c5f-95b0-81a2-b8d6-7dee662b049c(528072e2-5dae-18a2-f03c-a3ab99f6905f) -- TaskManager -- vim.TaskManager.createTask: :vmodl.fault.InvalidArgument

This is due to the incorrect permissions on vSAN health folder and files on vCenter. 

 vSAN health service log:

2025-05-14T06:23:52.737Z ERROR vsan-mgmt[1402576] [VsanVcDiskFormatConverterImpl::_Run opID=agw-0115827-5ae9] Disk conversion thread met unexpected error
Traceback (most recent call last):
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4201, in _Run
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4956, in ConvertDisks
  File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4385, in _RemediateBeforeConversion
  File "bora/vsan/clustermgmt/vpxd/VsanClusterPrototypeImpl.py", line 1252, in RemediateVsanCluster
  File "bora/vsan/clustermgmt/vpxd/VsanClusterPrototypeImpl.py", line 1268, in _RemediateVsanCluster
  File "bora/vsan/clustermgmt/vpxd/VsanClusterPrototypeHelper.py", line 28, in __init__
  File "bora/vsan/health/vpxd/pyMoVsan/VsanVcExtension.py", line 623, in CreateTaskWithTag
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in 
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
PyCppVmomi.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {
  msg = 'Received SOAP response fault from [<, /sdk>]: createTask\nA specified parameter was not correct: taskType',
  invalidProperty = 'taskType'

vpxd.log:

2025-05-14T06:23:52.729Z info vpxd[3650220] [Originator@6876 sub=vpxLro opID=agw-0115827-5ae9-de] [VpxLRO] -- BEGIN lro-67767942 -- TaskManager -- vim.TaskManager.createTask -- 52659c5f-95b0-81a2-b8d6-7dee662b049c(528072e2-5dae-18a2-f03c-a3ab99f6905f)
2025-05-14T06:23:52.736Z info vpxd[3650220] [Originator@6876 sub=vpxLro opID=agw-0115827-5ae9-de] [VpxLRO] -- FINISH lro-67767942
2025-05-14T06:23:52.736Z error vpxd[3650220] [Originator@6876 sub=Default opID=agw-0115827-5ae9-de] [VpxLRO] -- ERROR lro-67767942 -- 52659c5f-95b0-81a2-b8d6-7dee662b049c(528072e2-5dae-18a2-f03c-a3ab99f6905f) -- TaskManager -- vim.TaskManager.createTask: :vmodl.fault.InvalidArgument
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = ,
-->    invalidProperty = "taskType"
-->    msg = ""
--> }
--> Args:
-->
--> Arg obj:
--> 'vim.ClusterComputeResource:domain-c1065634'
--> Arg taskTypeId:
--> "com.vmware.vsan.clustermgmt.tasks.remediatevsancluster"
--> Arg initiatedBy:

The vSAN extension task "'com.vmware.vsan.clustermgmt.tasks.remediatevsancluster" information through the MOB may not be listed. 

This can be validated by accessing mob page, a window opens to request the vCenter credentials.

https://[VC_IP]/mob/moid=ExtensionManager&doPath=extensionList%5b%22com%2evmware%2evsan%2ehealth%22%5d%2etaskList

The issue is also related to the incorrect ownership of the vSAN health folder and files. They are owned by the user root instead of the user vsan-health. 

root@hicglxvc01 [ /etc/vmware-vsan-health ]# ls -lth
total 34M
-rwxrwxrwx 1 root root 6.2K Jun  5 05:06 vsanhealthstatuschange.json
-rwxrwxrwx 1 root root  12K Jun  5 04:42 vsan-releases.json
-rwxrwxrwx 1 root root  15M Jun  3 19:42 vsanvcg.db
drwxr--r-x 3 root root 4.0K Nov  7  2024 licenses
-rwxrwxrwx 1 root root 3.0K Nov  7  2024 logger.conf
-rwxr--r-x 1 root root  19M Nov  7  2024 vsanvcg.db.bak
-rwxrwxrwx 1 root root 339K Nov  7  2024 vsanhealthremediation.yaml
-rwxrwxrwx 1 root root 4.7K Nov  7  2024 vsanhosthealthmanifest.json
-rwxrwxrwx 1 root root 2.5K Nov  7  2024 cloudhealthmanifest.json
-rwxrwxrwx 1 root root 8.2K Nov  7  2024 vsanhealthcorrelation.json
-rwxrwxrwx 1 root root 4.3K Nov  7  2024 vsanhealth.properties
drwxr--r-- 3 root root 4.0K Nov  7  2024 cloudHealthResources
-rwxrwxrwx 1 root root  726 Nov  7  2024 config.conf
-rwxr--r-x 1 root root  573 Nov  7  2024 silent.json

Follow the steps below for the vSAN on-disk format version: 

Change the owner on the vSAN health folder, from vsan-health:root to vmware-vsan-health.

root@hicglxvc01 [ /etc ]# chown -R vsan-health:root vmware-vsan-health
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users logger.conf
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users licenses/
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users vsanhealthstatuschange.json
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users vsan-releases.json
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users config.conf
root@hicglxvc01 [ /etc/vmware-vsan-health ]# chown -R vsan-health:users silent.json

File permissions under the vSAN health folder were changed to vsan-health:users.

root@hicglxvc01 [ /etc/vmware-vsan-health ]# ls -lrth
total 34M
-rwxr--r-x 1 vsan-health users  573 Nov  7  2024 silent.json
-rwxrwxrwx 1 vsan-health users  726 Nov  7  2024 config.conf
drwxr--r-- 3 vsan-health root  4.0K Nov  7  2024 cloudHealthResources
-rwxrwxrwx 1 vsan-health root  4.3K Nov  7  2024 vsanhealth.properties
-rwxrwxrwx 1 vsan-health root  8.2K Nov  7  2024 vsanhealthcorrelation.json
-rwxrwxrwx 1 vsan-health root  2.5K Nov  7  2024 cloudhealthmanifest.json
-rwxrwxrwx 1 vsan-health root  4.7K Nov  7  2024 vsanhosthealthmanifest.json
-rwxrwxrwx 1 vsan-health root  339K Nov  7  2024 vsanhealthremediation.yaml
-rwxr--r-x 1 vsan-health root   19M Nov  7  2024 vsanvcg.db.bak
-rwxrwxrwx 1 vsan-health users 3.0K Nov  7  2024 logger.conf
drwxr--r-x 3 vsan-health users 4.0K Nov  7  2024 licenses
-rwxrwxrwx 1 vsan-health root   15M Jun  3 19:42 vsanvcg.db
-rwxrwxrwx 1 vsan-health users  12K Jun  5 04:42 vsan-releases.json
-rwxrwxrwx 1 vsan-health users 6.2K Jun  5 05:06 vsanhealthstatuschange.json

After changing the permissions, the "com.vmware.vsan.clustermgmt.tasks.remediatevsancluster" task should be visible in the vCenter MOB and will auto-register.