Poor Write Performance on PowerVault 220s Cluster

Question

I have seen several posts about poor write performance on the 220s in a cluster.   I have built my 4th cluster with a 220s and a pair of 2950s.  The previous clusters work and perform great.    This new one does not.  Simply copying a file from an array drive/folder to the same drive/folder yeilds horrible performance.  A 1 GB file takes about 3-4 minutes (and sometimes 15-20 minutes).  My other boxes can move the file within the same array drive/folder in 40 to 90 seconds.  Local RAID-1 array for OS can copy the same file to itself in about 10 sec.   Clusters are known to lose some performance for the redundancy they provide, but not this much.    All firmware and updates are installed (servers and 220s were just purchased 6 weeks ago).   Server Info:   PE 2950 / Dual Core 3.73 / PERC 4e/DC / 16GB RAM BIOS 1.1.0 (latest) ; PERC firmware 521X (latest)   Disk Array Info: PV 220s U320 / Dual EMMs / 13 300GB drives (5 Raid-1 arrays, 1 Raid-5 array) SES/EMM firmware is E.19 (latest) Read Policy=Adaptive Read Ahead Write Policy=Write Through Cache Policy=Direct I/O (these are dictated by clustering I believe and the same as my other clusters)   Cluster Info: Active/Active no errors during install or after, failover working fine   I have disabled PATROL READs on the PERC card as many other posts have suggested, but this did not help at all.   Anyone else with this issue have a solution?   Is it an issue with the PERC or SES firmware?  Can I roll these back without any data loss or issues?

Mark_A_Smith · Answer

Is this on a Raid 1 or Raid 5 array?  The Raid 5 array in the cluster is going to be slow....

Storagetalk · Answer

Putting a PERC into cluster mode kills the performance.  All the cache gets disabled.

smaybster · Answer

Yes...clusters are slower than a single server attached to a 220s because you give up the cache for redundancy/failover. Generally, you dont want mission critical stuff running with write cache anyways because of the loss of that cached data in the event of a system failure. These are larger 1GB writes beyond the size of the cache anyways (which I think is 128 or 256MB) so it would have to flush it to disk pretty quickly.

I'm just looking for similar/reasonable performance from a DELL-based cluster (as I can see on my 3 other 2850/220s clusters that are nearly identical where writes are 50-75% faster). This is a much faster server with 4 times the RAM of other servers and a newer (and supposed to be better) PERC 4e/DC controller. If it was my first cluster, then I may be inclined to say "that's just how it is".

Tests are run on the RAID-1 arrays (yes, RAID-5 has more write overhead).

Anyone out there with 2950's and 220s running active/active seeing this?

I think the problems are due to the fact that this is a 2950 (too new to have been truely tested in the real world). Even the first/last BIOS update brought a 50% gain in backup times.

pcmeiners · Answer

Two cents... Are the drive manufacturers the same, same model. Some array users state raid arrays will speed up if the  cache is turned off on the drives, which requires the manufacturers utility. Is the block size the same.

smaybster · Answer

All drives are identical.... (13) U320 Seagate 300GB ST3300007LC with the same rev number. I did not mess with cache on the drive or block size so these should all be the defaults.

David Hill DC · Answer

I have noticed the same performance problem in my clustered configuration. I currently have:

(2) Poweredge 6650
(2) Perc 4/DC
(1) Powervault 220s

Everything is running at U320 and I have the exact same drives you are using. The Seagate 300GB (ST3300007LC) with firmware D704. I've seen another post stating that the performance issues is due to the Seagate drives. I'm staring to believe this is true. I have some 74GB drives that I will install tomorrow, to see if that's the case.

smaybster · Answer

My performance issues were the result of a BAD CABLE.   Intially, the results of my tests indicated that the problem was on both nodes of the cluster.  Since the cluster was working fine, this seemed to point at the 220s or the fact that these were barnd new 2950s that maybe werent ready for prime time yet.   After further testing, I was getting good performance on one node having all the cluster resources failed over there, but not on the other.  After swapping EMMs and cable with the other node I noticed 1 of the 68 pins on one of the DELL cables to be shorter than the others.   This seemed to be the flaw that caused the drives to all report a negotiated speed of '10' instead of '320' to one of the nodes in the cluster which the DELL Tech initially diagnosed as a bug in the OpenManage and DSET report (since the negotiated speed data item was something new...previously it just shows a capable speed). The other node reported all virtual disks at '320' negotiated.  The controller showed '320' on both sides.   Just curious if your 'negotiated' speed on the virtual disks on both cluster servers is showing 320 in OM?   After a new cable arrived, now the performance is on pace with my other servers.   Would be interesting to hear results of your testing as well and whether there are also some performance issues with the seagate drives and what other special settings you found to improve performance of DELL clusters.

David Hill DC · Answer

I checked the negotiated and capable speed for the Seagate drives and they were listed at U320. So, I decided to test another RAID 5 array of Fujitsu 36GB drives, and much to my dismay, they also exhibited poor performance (2-3MB/s write, 20MB/s read).

After reading more on this forum, and reading some guides online about RAID performance, I decided to try a RAID 10 configuration. Thankfully, I have (6) 300GB drives, so this was an option. I configured the array (which is not very clear in PERC BIOS config) and ran my nBench performance benchmark again. This time performance was up to 8/MB write and 50-60/MB per second read. I've decided that will be enough for what I need, which is simply to run some virtual machines under MS Virtual Server R2.

PowerVault

Poor Write Performance on PowerVault 220s Cluster

Was this post helpful?