Troubleshooting Storage (iSCSI / Fibre Channel) issues with VMware vSphere ESX/ESXi.

Troubleshooting Storage (iSCSI / Fibre Channel) issues with VMware vSphere ESX/ESXi

To troubleshoot Storage (iSCSI / Fibre Channel) issues with VMware vSphere ESX/ESXi, perform the following steps:

1. How to determine if there is a network or fibre channel issue that could cause storage issues.
    
   Note: One should gather a vm-support bundle prior to any changes and have a good understanding of VMware. It is always a great idea to test changes in a development / test environment prior to making changes to production environments.
   • If the VM is moved to a different LUN, does performance improve?
   • If the number of VM's per LUN is reduced, does performance improve?
   • Are there SCSI errors in guest VM event log or messages?
   • Are all ESX/ESXi host experiencing the issue or just one host?
   • Is there a baseline of expected performance?
   • Does latency spike at a certain time or is it relatively constant?
   • Does the latency appear related to a VM or a particular LUN or iSCSI connection?
   • If the latency related to an iSCSI or Fibre Channel LUN, are other LUN's on the same storage device behaving OK?
   • If more than one storage device is connected via iSCSI or Fibre Channel, are both SAN devices experiencing the same issue?
      
2. Review Performance tab through VMware Client or use esxtop from ESX/ESXi cli
   • http://kb.vmware.com/kb/1008205
      
3. Review /var/log/vmkernel.log file or /var/log/messages depending on revision of ESX/ESXi and check for any wording containing "SCSI reservation conflict".
   • http://kb.vmware.com/kb/1005009
      
4. Segment or isolate iSCSI traffic from management and normal IO traffic.

• Equally distribute NIC ports installed on server for iSCSI traffic across redundant switches or fabric.

• Ensure iSCSI vmkernel ports have a one to one mapping with physical vmnic uplinks.
   
  Note: Best Practice would include two different iSCSI vSwitches for redundant iSCSI traffic. One could also have two vmkernel ports and two uplinks in the same vSwitch in a 1:1 mapping with a heartbeat connected to both uplinks. Reviewing the two different practices one can see the benefit and side effect of of each implementation.

• Disable Spanning Tree on all switch ports connecting to end devices and enable portfast setting on those switch ports, as these are endpoint devices, and cannot create fabric loops.

• If using Virtual Switch Tagging ( multiple or tagged VLAN's in ESXi vSwitches ), switch ports must use Trunk Mode.

• Determine VMware build revision installed and review release notes of newer VMware ESX/ESXi build version.

• Follow VMware and Dell best practices regarding applying driver / build updates as well as firmware updates.