One or More Storage Spaces Direct (S2D) Cluster Shared Volumes Become Detached or go Offline

Table of Contents

1. Description
2. Solution
3. Further Information

 

Note: This article is part of the Server Tutorials: Support Tools.

Description

In a multinode S2D cluster, when one or more nodes go down. Either planned by putting the node in maintenance mode or unplanned due to unexpected outage. One or more Cluster Shared Volumes (CSV) may become detached or offline. This is unexpected since the cluster is supposed to be able to handle the fault if it is within the fault tolerance design.

 

The Detached Operational Status can occur if the dirty region tracking (DRT) log is full. Storage Spaces uses DRT for mirrored spaces so that if a power failure occurs. Any in-flight updates to metadata are logged ensuring that the storage space can redo or undo operations to bring the storage space back into a flexible and consistent state, when power is restored and the system comes back up. If the DRT log is full, the virtual disk cannot be brought online until the DRT metadata is synchronized and flushed. This process requires running a full scan, which can take several hours to finish and is an offline process.

Solution

To determine if you are experiencing the DRT full issue that you can look at the additional DRT counters that are introduced in October Cumulative Update to ensure that the DRT log is not full before restarting it. There are two ways to verify this:

1. An Event ID 1023 is present in the Microsoft-Windows-StorageSpaces-Driver/Diagnostic event log where the NumberOfEntries event data equals the Limit event data. This indicates a DRT full scenario. An example DRT full, ID 1023 event:

Log Name:      Microsoft-Windows-StorageSpaces-Driver/Diagnostic

Source:        Microsoft-Windows-StorageSpaces-Driver

Event ID:      1023

Level:         Warning

Keywords:     

User:          SYSTEM

Computer:      nodename.domain.int

Description:

For internal use.

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Microsoft-Windows-StorageSpaces-Driver" Guid=" " />

    <EventID>1023</EventID>

    <Version>0</Version>

    <Level>3</Level>

    <Task>0</Task>

    <Opcode>0</Opcode>

    <Keywords>0x4000000000000000</Keywords>

    <EventRecordID>1394776</EventRecordID>

    <Correlation />

    <Execution ProcessID="9248" ThreadID="3820" />

    <Channel>Microsoft-Windows-StorageSpaces-Driver/Diagnostic</Channel>

    <Computer>nodename.domain.int</Computer>

    <Security UserID="S-1-5-18" />

  </System>

  <EventData>

    <Data Name="SpaceId"> </Data>

    <Data Name="NumberOfEntries">386</Data>

    <Data Name="Limit">386</Data>

  </EventData>

</Event>

2. Another way to find out if DRT is full is to check if the Synchronizing Count in the Perfmon counters equals the Limit value.  In the example below, the Synchronizing Count (0) does not equal the Limit (255).  The DRT is not full in this case.

Further Information

Dell and Microsoft recommend applying the fix which is publicly available via KB4480977 (https://support.microsoft.com/en-us/help/4480977/windows-10-update-kb4480977).

Note: If the DRT log is full, please follow the instructions on running the Data Integrity Scan and Storage Maintenance Mode to shutdown/restart nodes as described in KB4294480 and in the S2D Troubleshooting doc.

Note: see the attached document for details on patch updating.