VxRail: VSAN drives 'Not mounted on this host' and 'Ineligible for use by VSAN' cannot be added to VSAN disk groups

After checking the disks with the vdq -qH or vdq -Hi commands, one or more drives show Reason: Not mounted on this host. These, and some others show State: Ineligible for use by VSAN. Among those that are Ineligible, there may be Boss cards, Satadom, RecoverPoint disks, and other types that should show that state. Within vCenter, more drives than expected may be seen, though the extra ones are not available to add to disk groups.

New drives may not be visible after being added or replaced following a drive failure. Drives that have nothing after Name: should be naa.<numbers&letters>. These are commonly called ghost disks or phantom drives. They are in a place where a now-lost device used to be. Having ghost disks present can result in issues such as long boot times, failing validations, inability to ensure accessibility, and host crashes.

Removing ghost drives is often possible through vCenter or by command line. However, sometimes they fail to remove, and you often have to fix the Not mounted… drive first.

If the VSAN detects a failure to write to a drive, it may remove it from the VSAN. This may occur even if the hardware sensors have not seen a condition that would cause them to mark the disk as faulted. If that disk is a cache drive or if using deduplication and compression, the VSAN must take the entire disk group offline. While this can lead to the above-mentioned conditions, it is not the underlying cause. The cause is corrupted metadata or disks that still have partitions (from their former configuration). They are not recovered and ready to be added back to the VSAN. This can also occur for other reasons when something is inadvertently written over disk metadata improperly. The data is intact but no longer accessible, and the VSAN must recover storage policy compliance with a resynchronization.

A drive with this partition type may think it is part of a disk group and show a cache drive where there should not be one. This cache drive does not have normal information like the capacity or name (Network Adapter Address (NAA) information missing). You cannot remove it, however, because the host thinks there is a drive there which is not mounted. You are also unable to correct this by re-scanning the storage controllers (this can cause a host crash) or by rebooting the host.

Fix: Any Not mounted… drives must have their partitions removed or hidden and any ghost disks must be removed from the environment. If partitions are masked, this should still allow them to show up as Eligible for use by VSAN again. Adding them to a disk group should wipe anything that was on them during the process. After fixing that, and removing any ghost disks, you may need to reboot the host. This is done after everything appears properly on the host. In vCenter's Cluster > Configure > Disk Management area, you can create a disk group as normal.

Steps:
Place the host into Maintenance Mode (Ensure Accessibility). This protects data on the host from any mistakes or unexpected issues. Ensure that the rest of the VSAN is healthy. If a VSAN resynchronization is in process, this must complete before any disks or disk groups with data on them can be removed from the VSAN.
Broadcom has introduced a simple Erase ESXi storage Devices feature in versions 7.X and later.
Erase ESXi Storage Devices (vmware.com)

1. In the vSphere Client, go to the ESXi host.
2. Click the Configure tab.
3. Under Storage, click Storage Devices.
4. Select one or more devices and click the Erase Partitions icon.
5. Verify that the partition information you are erasing is not critical.
6. Click OK to confirm your change.

If the above does not work, use the manual steps below:

1. Run the below command on the host (in PuTTY) and copy the output to a document. PuTTY is not required but being able to copy and paste is helpful.

   vdq -qH

2. Identify drives that are Ineligible for Use by VSAN AND either show Reason: Not mounted on this host or have nothing in the Name field (no NAA).

3. Correct drives showing Not mounted… first:

   1. Get the NAA of the ineligible disk from the vdq -qH output on the host. Run this command to mask the partitions on the disk:

      partedUtil mklabel /dev/disks/<naa.#'s> gpt

   2. Run the following command again and ensure that the drive now shows Eligible for use by VSAN.

      vdq -qH

      If not, a reboot and then repeat of the previous step is needed. Remove ghost disks before rebooting to avoid a long boot process as the host initializes disks and vSAN services attempt to start.

4. Remove ghost disks. You can usually do this in the same Disk Management area. If not, use the host command line.

   $ esxcli vsan storage remove -u <UUID>

   Make note of the Universally Unique Identifier (UUID) of disks without NAA names, from the Step 1 output.

5. Check that everything looks as it should. Refresh vCenter and check Disk Management again and run the vdq -qH command on the host to ensure that all expected drives appear and show Eligible for use by VSAN. If not, reboot the host as some drives may not have been initialized yet and check again.

6. Create the disk group or add disks to existing disk groups as normal (if using deduplication and or compression, full disk group recreation is needed).