Unsolved
This post is more than 5 years old
2 Posts
0
6975
December 30th, 2003 20:00
PERC3/PERC4 Write Performance: Working as Intended
We purchased four dual-2.8Ghz 2650s, with PERC3/DC cards, along with two PowerVault 220S arrays (Ultra320 version), to create two storage clusters as recommended by Dell Power Solutions and Red Hat Enterprise.
Cluster testing went relatively smoothly. Failover worked under both RHEL AS 2.1 and 3.0. However, we noticed writes to the shared storage were slower than expected. So, we went back to a basic configuration with factory defaults to diagnose the problem:
With a single 36gb-10K-U320 disk in the PV220S, connected to one 2650, on RedHat 9 installed by the Open Manage CD (v7.5), no clustered SCSI, a sequential write could be done at 20 MB/s.
Changing to "WRITETHRU" mode on a single drive, required by clustering, sequential writes dropped to 10 MB/s. On a 5-disk RAID5, writes fell to 8 MB/s.
In comparison, we can write to a PE2650 single internal disk, with the on-board Adaptec AIC-7899, at 35 MB/s:
sync; time dd ibs=1048576 obs=1048576 count=1024 if=/dev/zero of=/var/gigfile;
1024+0 records in 1024+0 records out
real 0m25.274s user 0m2.690s sys 0m11.730s
time sync
real 0m3.645s user 0m0.000s sys 0m0.040s
(1024 MB/28.9 seconds = 35.4 MB/s)
We tried the following to improve PERC3/DC writes, with little effect:
* Check logs and settings:
No failures or events in the BIOS, "messages", or megamon/raid monitor.
Disk, channel and "megaraid" all report U160 speed, proper termination.
* Change the defaults on the PERC3/DC (to):
Write Policy = WRBACK (WTHRU dropped peformance by 40%)
Read Policy = ADAPTIVE (tried NONE, READ AHEAD)
Cache Policy = DIRECTIO (tried CACHEDIO, which gave a small gain)
BIOS: Enabled (DISABLED, required by clustering)
Stripe size: 64k (others had no effect)
* Update all BIOS and Firmware:
PV220S revision E.14, Ultra320 Version
PE2650 version A15 through A17
PERC3/DC Dell firmware 1.92 (BIOS 3.31)
(also tried LSI Elite 1600 Firmware C170 through 111Q)
* Update software Megaraid driver
Tried versions 1.18h, 1.18j and the latest 2.00.9
* Change OS:
Tested with RedHat AS 2.1, RedHat 9.0, RedHat AS 3.0.
* Change disk models:
Atlas U320 IV 10K 36Gb are included. Also tried IBM Ultrastar U320 10K 36Gb.
* Change channels and cables:
Configured PV220S on Channel 1 instead of 0. New LVD cables tried.
* Change slots:
Moved the PERC3/DC from slot 1 (66mhz shared) to slot 3 (133mhz dedicated).
* Change enclosures and bus modes:
Tested with a second PV220S in shared, split-bus and clustered modes.
Also tried with the PowerEdge internal U160 backplane. Same results.
* Change PowerEdges:
Two seperate PowerEdges tested individually.
After speaking with several Dell technicians, and escalating the issue, we exchanged the PERC3/DC's for PERC4/DC's.
With a PERC4/DC, factory-defaulted sequential writes to a single disk (without writethrough), went from 20 MB/s to nearly 40 MB/s. However, just by switching to "writethrough", they dropped to 10 MB/s (8 MB/s on RAID5).
An Advanced Software Support technician was then contacted, who was able to forward our results to the Dell Hardware Escalation team. They concluded the following:
"The server and power vault are working as designed and we do not expect a solution to the performance difference on this configuration."
We hope that Dell chooses to more truthfully advertise the performance tradeoff of the cluster configuration they recommend. If more accurate information were to be provided prior to sale, customers might no longer be suprised by this issue.
Cluster testing went relatively smoothly. Failover worked under both RHEL AS 2.1 and 3.0. However, we noticed writes to the shared storage were slower than expected. So, we went back to a basic configuration with factory defaults to diagnose the problem:
With a single 36gb-10K-U320 disk in the PV220S, connected to one 2650, on RedHat 9 installed by the Open Manage CD (v7.5), no clustered SCSI, a sequential write could be done at 20 MB/s.
Changing to "WRITETHRU" mode on a single drive, required by clustering, sequential writes dropped to 10 MB/s. On a 5-disk RAID5, writes fell to 8 MB/s.
In comparison, we can write to a PE2650 single internal disk, with the on-board Adaptec AIC-7899, at 35 MB/s:
sync; time dd ibs=1048576 obs=1048576 count=1024 if=/dev/zero of=/var/gigfile;
1024+0 records in 1024+0 records out
real 0m25.274s user 0m2.690s sys 0m11.730s
time sync
real 0m3.645s user 0m0.000s sys 0m0.040s
(1024 MB/28.9 seconds = 35.4 MB/s)
We tried the following to improve PERC3/DC writes, with little effect:
* Check logs and settings:
No failures or events in the BIOS, "messages", or megamon/raid monitor.
Disk, channel and "megaraid" all report U160 speed, proper termination.
* Change the defaults on the PERC3/DC (to):
Write Policy = WRBACK (WTHRU dropped peformance by 40%)
Read Policy = ADAPTIVE (tried NONE, READ AHEAD)
Cache Policy = DIRECTIO (tried CACHEDIO, which gave a small gain)
BIOS: Enabled (DISABLED, required by clustering)
Stripe size: 64k (others had no effect)
* Update all BIOS and Firmware:
PV220S revision E.14, Ultra320 Version
PE2650 version A15 through A17
PERC3/DC Dell firmware 1.92 (BIOS 3.31)
(also tried LSI Elite 1600 Firmware C170 through 111Q)
* Update software Megaraid driver
Tried versions 1.18h, 1.18j and the latest 2.00.9
* Change OS:
Tested with RedHat AS 2.1, RedHat 9.0, RedHat AS 3.0.
* Change disk models:
Atlas U320 IV 10K 36Gb are included. Also tried IBM Ultrastar U320 10K 36Gb.
* Change channels and cables:
Configured PV220S on Channel 1 instead of 0. New LVD cables tried.
* Change slots:
Moved the PERC3/DC from slot 1 (66mhz shared) to slot 3 (133mhz dedicated).
* Change enclosures and bus modes:
Tested with a second PV220S in shared, split-bus and clustered modes.
Also tried with the PowerEdge internal U160 backplane. Same results.
* Change PowerEdges:
Two seperate PowerEdges tested individually.
After speaking with several Dell technicians, and escalating the issue, we exchanged the PERC3/DC's for PERC4/DC's.
With a PERC4/DC, factory-defaulted sequential writes to a single disk (without writethrough), went from 20 MB/s to nearly 40 MB/s. However, just by switching to "writethrough", they dropped to 10 MB/s (8 MB/s on RAID5).
An Advanced Software Support technician was then contacted, who was able to forward our results to the Dell Hardware Escalation team. They concluded the following:
"The server and power vault are working as designed and we do not expect a solution to the performance difference on this configuration."
We hope that Dell chooses to more truthfully advertise the performance tradeoff of the cluster configuration they recommend. If more accurate information were to be provided prior to sale, customers might no longer be suprised by this issue.
No Events found!