VMware ESXi: Troubleshooting Storage issues within VMware vSphere ESXi

Troubleshooting Storage (iSCSI/Fibre Channel) issues with VMware vSphere ESX/ESXi

1. How to determine if there is a network or fiber channel issue that could cause storage issues.
    
   Note: One should gather a vm-support bundle prior to any changes and have a good understanding of VMware. It is always a great idea to test changes in a development/test environment prior to changing production environments.
   • If the VM is moved to a different LUN, does performance improve?
   • If the number of VMs per LUN is reduced, does performance improve?
   • Are there SCSI errors in guest VM event log or messages?
   • Are all ESX/ESXi hosts experiencing the issue or one host?
   • Is there a baseline of expected performance?
   • Does latency spike at a certain time or is it relatively constant?
   • Does the latency appear related to a VM or a particular LUN or iSCSI connection?
   • If the latency is related to an iSCSI or Fibre Channel LUN, are other LUNs on the same storage device behaving OK?
   • If more than one storage device is connected using iSCSI or Fibre Channel, do both SAN devices experiencing the same issue?
      
2. Review Performance tab through VMware Client or use esxtop from ESX/ESXi cli
   • https://knowledge.broadcom.com/external/article?legacyId=1008205
      
3. Review /var/log/vmkernel.log file or /var/log/messages depending on revision of ESX/ESXi and check for any wording containing "SCSI reservation conflict." 
4. Segment or isolate iSCSI traffic from management and normal IO traffic.

• Equally distribute Network Interface Card (NIC) ports installed on server for iSCSI traffic across redundant switches or fabric.
• Ensure that iSCSI VMkernel ports have a one to one mapping with physical vmnic uplinks.
   
  Note: Best Practice would include two different iSCSI vSwitches for redundant iSCSI traffic. One could also have two VMkernel ports and two uplinks in the same vSwitch in a 1:1 mapping with a heartbeat connected to both uplinks. Reviewing the two different practices one can see the benefit and side effect of each implementation.
• Disable Spanning Tree on all switch ports connecting to end devices and enable PortFast setting on those switch ports, as these are endpoint devices, and cannot create fabric loops.
• If using Virtual Switch Tagging (multiple or tagged VLANs in ESXi vSwitches), switch ports must use Trunk Mode.
• Determine VMware build revision installed, and review release notes of newer VMware ESX/ESXi build version.
• Follow VMware and Dell best practices regarding applying driver and build updates and firmware updates.