upgraders
2 Bronze

3 Identical R710's PERC H700 but one has VERY slow Disk IO

I am baffeled as I am setting up a new 3 Node cluster with three totally identical R710's

6 Core Dual Xeon

(2) 300GB 10K SAS in a RAID 1 for the OS  (C DRIVE)

(6) 900GB 10K SAS in RAID 0 storage  (D DRIVE)

All drive are DELL Drives and all the same type

H700 with 1GB nVRAM

144GB Ram

All the servers have updated Firmware on ALL the drives, PERc H700, BIOS etc..  I run the Server setup that does the checkign and I manually checked the firmware is the same on all the machines.

All three are running Windows 2012 R2.

So the problem is that I am just using rudimentary testing by doing a single file copy from each of the Virtual Drives of a 5GB file and a 12GB file. Nothing Fancy, but the whole reason I am doing this is because Veeam is reporting latency on the drives on one of the servers.

So I have Node1, Node2 and Node3

Node1 & Node2 are clustered and have been migrating VM's to to. So far Node2 is not sharing any loads, but I was about to bring on Node3, before some investigation work. 

Node1:

If I copy a 5GB files from D drive to D drive I get about 600Mbps steady

Copy C to C about 950Mbps

Copy from D to C is about 900Mbps steady

Copy from C to D is about 600Mbps steady

Node2:

If I copy a 5GB files from D drive to D drive I start off at 600MBPS and then it drops down to 200Mbps steady

Copy for C to C 20Mbps to 30Mbps (that is correct VERY VERY SLOW)

Copy from D to C starts off at around 400Mbps and quickly drops down to about 150Mbps

Copy from C to D  Is 110Mbps steady

Node3:

If I copy a 5GB files from D drive to D drive I get about 700Mbps steady (not using this storage at all)

Copy C to C is 960Mbps

Copy from D to C is about 980Mbps steady

Copy from C to D is about 600Mbps steady

So you can clearly see something is going on with Node 2.  There is nothing going on with the drives, very low "resting" state.  All my VMs are running on Node 1 so there is nothing that this server is doing yet. 

I am at a loss on what to look for.  The Dell logs are clean, no errors or anything.  The other two Node (1 and 3) do NOT have Cache enabled with these numbers.  On Node 2 I turned it ON and there was no performance change.

Each Virtual Disk has Adaptive Read Ahead and Write back policy, although they do not have the Disk Cache Policy turned on (Which I plan to do when I figure out this problem)

The H700 has the current Firmware:


Driver Version 6.600.21.08
Storport Driver Version 6.3.9600.17090

Does anyone have a Clue as what to check?  The Windows 2012 R2 is the same server configuration load and Sysprepped and both have the same basic software so I am clueless. 

Things I have tested and tried:

Turing off the Windows Caching (although the OS settings are Identical)

Changing the PERC to Force Writeback, toggling Cache on and off.

Thanks for ANY guidance with this

Jason

 

0 Kudos
7 Replies
DELL-Bo Pham
Moderator
Moderator

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Jason, 

Thanks for the detail testing you've done. You have ruled out a lot of the possibilities.  My first thought was a failed battery forcing Write-Through as a possibly, but with you setting it to Force Write-Back, that rules it out. 

Since everything configuration wise appears to be equal, do you mind exporting the PERC controller log from all 3 Nodes and label them Node1, Node2, & Node3.  Please private message me the logs and i'll look them over for any clues.

You've definitely got a strange one and we'll be glad to see if we help figure it out. 

Bo Pham

Dell | Social Outreach Services - Enterprise Quality Lead
Get Support on Twitter @DellCaresPRO 

 

0 Kudos
upgraders
2 Bronze

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Thanks, which log reports do you want. Looks like there is a few.
0 Kudos
upgraders
2 Bronze

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Never mind I was looking at the reports.  I have all three Logs Zipped up, but after 15 Minutes I can not see how I can PM you without being a "friend"  

THANKS!!

Jason

0 Kudos
upgraders
2 Bronze

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Here are the Logs <snip>

0 Kudos
DELL-Bo Pham
Moderator
Moderator

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Sorry about the attachment issues Jason. I have sent you an email so that you can send me the logs for review.

Bo Pham

Dell | Social Outreach Services - Enterprise Quality Lead
Get Support on Twitter @DellCaresPRO 

 

0 Kudos
upgraders
2 Bronze

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

I thought I would provide closure to this thread, only because I hate when I find a post somewhere and you never know what the resolution was.  In my case, I simply replaced the card and then all was fine.  I never could find out why and Bo was not able to find anything (Thanks for looking by the way) 

0 Kudos
DELL-Bo Pham
Moderator
Moderator

RE: 3 Identical R710's PERC H700 but one has VERY slow Disk IO

Thanks for posting the update. Good to hear that the replacement resolved the issue.  It was indeed very odd that all the logs seemed fine.  Great working with you on this and i'm glad we got to give it a shot at least. 

Bo Pham

Dell | Social Outreach Services - Enterprise Quality Lead
Get Support on Twitter @DellCaresPRO 

 

0 Kudos