| Article Summary: |
This article provides information on possible cause and how to troubleshoot when an ESX/ESXi host locks a series of on-disk resource metadata structures delaying operations from other hosts. |
Symptoms:
-
ESX/ESXi host locks a series of on-disk resource metadata structures, resulting in delays for operations from another host.
-
This article only applies when the "type 10c00002" is seen in the error message.
-
In the vmkernel.log file (located at /var/log) of the hosts that are being delayed, you see entries similar to:
DLX: 3394: vol 'datastore-name': [Req mode 1] Checking liveness of [type 10c00002 offset 23714816 v 8, hb offset 3903488
gen 53, mode 1, owner 50a3da5a-a3b687f2-9b6a-842b2b0c3b60 mtime 255486 nHld 0 nOvf 0]
DLX: 3901: vol 'datastore-name': [Req mode: 1] Not free; Lock [type 10c00002 offset 23714816 v 8, hb offset 3903488
gen 53, mode 1, owner 50a3da5a-a3b687f2-9b6a-842b2b0c3b60 mtime 255486 nHld 0 nOvf 0]
Cause:
During file related activities on a VMFS datastore, the ESX/ESXi host acquires a lock on a series of on-disk resource metadata structures to either allocate file blocks to a file or free file blocks while deleting the file from VMFS datastore. After it completes the resource metadata structure related operations, the ESX/ESXi host continues to retain the lock for a short period of time in case the host gets further requests for allocations. In an ESX/ESXi cluster where multiple hosts are actively working with the same VMFS datastore, this lock from one host can delay operations from another.
Under most conditions the delay is unnoticed operationally. However, the chance of this delay becoming apparent to user level operations increases under these conditions:
- Large number of hosts (>10) in a cluster attempting to allocate files on VMFS datastores with low % free space (< 10%).
- Power-off or power-on operations on many virtual machines in a short period of time.
- vMotion of virtual machines that involves VMFS datastores with low % free space (< 10%).
- High activity on disk trees that contain a lot of linked clones.
- High rate of growth with thin disk provisioning.
Resolution:
The issue is resolved in:
Reference:
ESX/ESXi host locks a series of on-disk resource metadata structures delaying operations from other hosts.