Checking VMFS metadata consistency using vSphere On-disk Metadata Analyzer (VOMA)
Summary: The VOMA tool can be used to perform a file system check of metadata.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
- When there are problems accessing certain files on a VMFS datastore
- When it is not possible to modify or erase files on a VMFS datastore
- When attempting to read files on a VMFS datastore, it fails with an error
Invalid argument
When attempting to modify the partition table, fails with error:
Error: Read-only file system during write on /dev/disks/naa.600601603b503400de93e0a87e9ce211 SetPtableGpt: Unable to commit to disk
In the /var/log/vmkernel file can be seen entries similar to:
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
Cause
The following scenarios would apply to perform a metadata check:
- The environment experiences storage outage.
- After a RAID volume rebuild or a disk replacement
- There are metadata errors in the vmkernel.log file.
- It is unable to access files on a VMFS datastore that is not present on any other host.
Resolution
Prerequisites:
Before running VOMA, it is required to ensure that:
- Power off all virtual machines on the affected datastore or migrated to another datastore.
- Shutting down a virtual machine running on files having certain types of corrupt metadata by the vSphere On-disk Metadata Analyzer, may make the virtual machine and its data permanently unavailable. Because of this, it is always advisable to have current backups of the virtual machines in the environment.
- For VMFS-5/VMFS-6 datastores, the datastore is unmounted on all ESXi 7.x/8.x hosts.
- If VOMA is run against a volume with access to one or more ESXi hosts, VOMA produces a message similar to:
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000: Checking if device is actively being used by other hosts Found 1 actively heart beating hosts on device '/vmfs/devices/disks/naa.00000000000000000000000000:1' 1): MAC Address xx:xx:xx:xx:xx:xx
- If VOMA is run against a volume with access to one or more ESXi hosts, VOMA produces a message similar to:
To check metadata consistency, run VOMA from the CLI of an ESXi host. VOMA can be used to check and fix minor inconsistency issues for a VMFS datastore or logical volumes that back the VMFS datastore.
VOMA can check and fix the following items.
| VOMA Functions | Description |
|---|---|
Metadata check and fix |
Examples of metadata check and fix include, but are not limited to, the following:
|
Affinity metadata check and fix |
To enable the affinity check for VMFS6, use the -a|--affinityChk option.Several examples of affinity metadata check and fix include the following:
|
Directory validation |
VOMA can detect and correct the following errors:
|
Lost and found files |
During a file system check, VOMA can find files that are not referenced anywhere in the file system. These orphaned files are valid and complete, but do not have a name or directory entry on the system. If VOMA encounters orphaned files during scanning, it creates a directory named lost+found at the root of the volume to store the orphaned files. The names of the files use the File sequence number format. |
Command options that the VOMA tool takes include the following.
| Command Option | Description |
|---|---|
-m|--module |
Available modules are:
|
-f|--func |
Functions the module support:
|
-a|--affinityChk |
Include an affinity-related check/fix for VMFS6. |
-d|--device |
Device and Disk to be used. For example: voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x |
-b|--blockSize |
Indicate the disk block size. |
-s|--logfile |
Specify the path to the log file to output the results. |
-x|--extractDump |
Extract the collected dump using VOMA. |
-D|--dumpfile |
Indicate the dump file to save the collected metadata dump. |
-v|--version |
Display the version of VOMA. |
-h|--help |
Display the help message for the VOMA command. |
-Y |
Indicate that VOMA run without using PE tables for address resolution. |
-Z| --file |
Indicate that VOMA run on extracted device files. |
Procedure:
Obtain the name and partition number of the device that backs the VMFS datastore that you want to check.
#esxcli storage vmfs extent list
The Device Name and Partition columns in the output identify the device. For example:
Volume Name ..... Device Name Partition 1TB_VMFS6 ..... naa.xxxx 3
Check for VMFS errors.
Provide the absolute path to the device partition that backs the VMFS datastore, and provide a partition number with the device name. For example:
# voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x The output lists possible errors. For example, the following output indicates that the heartbeat address is invalid. XXXXXXXXXXXXXXXXXXXXXXX Phase 2: Checking VMFS heartbeat region ON-DISK ERROR: Invalid HB address Phase 3: Checking all file descriptors. Phase 4: Checking pathname and connectivity. Phase 5: Checking resource reference counts. Total Errors Found: 1
Reference documentation:
Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (broadcom.com)
Checking Metadata Consistency with VOMA (vmware.com)
Affected Products
VMware ESXi 7.x, VMware ESXi 8.xArticle Properties
Article Number: 000226655
Article Type: Solution
Last Modified: 28 Jan 2025
Version: 2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.