Cluster storage loses access to cluster shared volumes

Issue:

Event ID 5120 "Cluster Shared Volume 'Volume1' ('Volume1') is no longer available on this node because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished." is generated on a Windows server that is a member of a Windows Server 2012 or Windows Server 2012 R2 failover cluster. All connectivity is lost to the LUN on the SAN. Access to the cluster shared volume is no longer available.

Solution:

1. Ensure all network interface card (NIC) drivers and firmware are up to date.

2. Ensure all fiber host bus adapters (HBA) drivers and firmware are up to date.

3. Update the network or fiber switches to the latest firmware.

4. Update or remove any network threat protection or firewall applications installed on the server.

5. Apply the recommended hotfixes for failover clusters for Windows Server 2012. The latest hotfix at the time this solution was created can be found here.

6. Apply the recommended hotfixes for failover clusters for Windows Server 2012 R2. The latest hotfix at the time this solution was created can be found here.

 

Additional Information:

This information applies to Windows Server 2012 and Windows Server 2012 R2.

Event ID 5120 is logged in the System event log.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 4/12/2014 10:00:43 PM
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:     
User: SYSTEM
Computer: ComputerName.domain.local
Description:
Cluster Shared Volume 'Volume1' ('Volume1') has entered a paused state because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

The following entry is logged in the Cluster log.

0000083c.00000d40::2014/04/13-09:02:00.958 INFO  [DCM] CsvFs event CsvFsVolumeStateChangeFromIO for volume Volume1:a205e5b2-f41b-45ce-9b45-7a00df23a590, status STATUS_IO_TIMEOUT(c00000b5)

The following Event ID's are often associated with Event ID 5120.

Event ID 1230

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 4/12/2014 10:00:43 PM
Event ID: 1230
Task Category: Resource Control Manager
Level: Error
Keywords:     
User: SYSTEM
Computer: ConputerName.domain.local
Description:
A component on the server did not respond in a timely fashion. This caused the cluster resource 'ResourceName' (resource type 'Virtual Machine', DLL 'vmclusres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.

Event ID 1069

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 4/12/2014 10:00:43 PM
Event ID: 1069
Task Category: Resource Control Manager
Level: Error
Keywords:     
User: SYSTEM
Computer: ConputerName.domain.local
Description:
Cluster resource 'VitrualMachineName' of type 'Virtual Machine' in clustered role 'VirtualMachineName' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it.  Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.