I'm noticing a lot of spikes on our Equallogic iscsi storage, when I go to the advanced>disk/Virtual disk monitor, we seem to be getting a lot of spikes at ranging from 22-160 ms.  The average numbers look good, 2ms and such.  But in the course of a few minutes we'll have a number of these (usually) read latencies on a number of vm's on different hosts. I fired up esxtop and looking at disk view under the DAVG i'm seeing a few of my luns hitting again anywhere from 14 all the way up to 166.  It's not constant but I can usually find a few with in a few minute period.  This seems to be on a few different SAN's.  But before i go opening support cases I just wanted to see if this is normal behavior and I'm making a mountain out of a mole hill or if this warrants further investigation.  

* Our sans are all a few firmware version behind except one.  And it's having similar issues, though not as extreme.  We'll be updating the controllers over the weekend.

* We are using the EQL MEM, SAN HQ is showing that everything is fine and not generating any alarms and our IOP counts are low/medium. 

* I've been over the EQL best practices guide a few times and verified that we do have everything configured correctly for that document.  

* our NIC fimrware and drivers all appear to be up to date on the hosts

* we do notice a performance problem on the servers a lot of times, they can be very sluggish when you are RDP'd into them. 

* I'm having our network guy check the switches this traffic passes through, but they are saying all their metrics look good.  Our entire network is 10 gig with the iscsi on it's own dedicated NIC's/Switches

*  Weirdly though, the average rate for the iscsi data is only 10-15 mb's.  At least what ops manager is reporting on the dvs.