VxRail: A single failed NVMe disk causes the entire VSAN cluster to fault with IO errors

Summary: A single failed NVMe disk causes the entire vSAN cluster to fault with IO errors.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • VxRail 7.x code version
  • NVMe disk failure
  • PDL events are reported on hostd.log
2024-05-23T04:49:18.562+0100 info hostd[61598519] [⋮ sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
-->    key = 135,
-->    chainId = -1,
-->    createdTime = "1970-01-01T00:00:00Z",
-->    userName = "",
-->    host = (vim.event.HostEventArgument) {
-->       name = "host.domain.com",
-->       host = 'vim.HostSystem:ha-host'
-->    },
-->    eventTypeId = "esx.problem.vob.vsan.pdl.offline",
-->    arguments = (vmodl.KeyAnyValue) [
-->       (vmodl.KeyAnyValue) {
-->          key = "1",
-->          value = "52071875-618f-3f4b-27f5-89ab5d2a9bf6"
-->       }
-->    ],
-->    objectId = "ha-host",
-->    objectType = "vim.HostSystem",
--> }
 
  • vSAN management logs reports "Stuck Descriptor" messages
2024-05-23T04:49:09.355+0100 cpu99:2100019)DOM: DOM2PCPrintDescriptor:2121: [1287682095:0x45dabbe1f140] => Stuck descriptor
2024-05-23T04:49:10.942+0100 cpu122:2100017)DOM: DOM2PCPrintDescriptor:2121: [11772501:0x45dabbf65d40] => Stuck descriptor
2024-05-23T06:02:49.344+0100 cpu73:2100015)DOM: DOM2PCPrintDescriptor:2121: [30274285827:0x45dabbf2a840] => Stuck descriptor
 
  • VMkernel reports stuck IO
2024-05-23T04:49:09.379+0100 cpu43:2099914)DOM: DOM2PCPrintDescriptor:2121: [14235787:0x45bac61c17c0] => Stuck descriptor
2024-05-23T04:50:21.899+0100 cpu20:67978583)ScsiDeviceIO: 12480: Task mgmt request issued to device t10.NVMe____Dell_Ent_NVMe_CM6_RI_7.68TB_____________0XXXXXXXXXEE38C is stuck (WorldID 0,
md 0x28, CmdSN 7daf4). Issuing yellow notification to the
2024-05-23T04:50:21.899+0100 cpu20:67978583)ScsiDeviceIO: 12559: FDS_DEV_EVENT_REPORT_STUCK_IO event for device t10.NVMe____Dell_Ent_NVMe_CM6_RI_7.68TB_____________016CD616E28EE38C
2024-05-23T04:51:33.949+0100 cpu64:67978582)ScsiDeviceIO: 12527: Task mgmt request issued to device t10.NVMe____Dell_Ent_NVMe_CM6_RI_7.68TB_____________0XXXXXXXXXEE38C is stuck (WorldID 0,
md 0x28, CmdSN 7daf4). Issuing red notification to the
2024-05-23T04:51:33.949+0100 cpu64:67978582)ScsiDeviceIO: 12559: FDS_DEV_EVENT_REPORT_STUCK_IO event for device t10.NVMe____Dell_Ent_NVMe_CM6_RI_7.68TB_____________0XXXXXXXXXEE38C

Cause

  • Quiesce operations do not complete if there are pending IOs on the disk.
  • If there is stuck IOs, IOs are never expected to complete sometimes if the driver could not complete the IOs.
  • With STUCK IOs at device layer, and if a quiesce operation is initiated due to transient error handling or APD handling or any DECOM operation, the quiesce never completes as cleanup does not proceed due to pending STUCK IOs.
  • This causes a race condition.

 

Resolution

Issue is addressed in ESXi 7.0U3 P09 or latest, which is in VxRail code release version 7.0.520.

Workaround if the host is impacted:

  1. Unregister the virtual machines on the nodes
  2. Put host in maintenance mode
  3. Reboot host.
  4. Replace NVMe disk if disk reports HW failure.
  5. Evaluate the upgrade as below for a permanent fix.

Affected Products

VxRail, VxRail Appliance Series, VxRail Software
Article Properties
Article Number: 000225946
Article Type: Solution
Last Modified: 11 Apr 2025
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.