Start a Conversation

Unsolved

This post is more than 5 years old

E

16895

February 11th, 2010 07:00

PS3900XV performance issues?

We've had a PS3900XV connected to some Linux servers for a couple of years now and we still can't seem to get decent performance out of it.  Our configuration is:

Dell 1950 Dual 2.6GHz X5355 Quad Cores with 20GB of RAM running Centos 5.4
Qlogic 4062 HBA
Pair of Dell 6224 switches configured as recommended by Dell for redundancy
PS3900XV

Here are some performance tests I just sent to Dell on an open service call (been open for over a year now):

I did the following just now on our system (which is in use and running jobs):

[root@merge01 ~]# hdparm -T -t /dev/mapper/pasmerges; hdparm --direct -T -t /dev/mapper/pasmerges

/dev/mapper/pasmerges:
 Timing cached reads:   9168 MB in  2.00 seconds = 4588.60 MB/sec
 Timing buffered disk reads:  148 MB in  3.02 seconds =  49.05 MB/sec

/dev/mapper/pasmerges:
 Timing O_DIRECT cached reads:   448 MB in  2.01 seconds = 223.16 MB/sec
 Timing O_DIRECT disk reads:  310 MB in  3.00 seconds = 103.33 MB/sec

[root@merge01 ~]# hdparm -T -t /dev/mapper/pasmerges; hdparm --direct -T -t /dev/mapper/pasmerges

/dev/mapper/pasmerges:
 Timing cached reads:   9552 MB in  2.00 seconds = 4780.38 MB/sec
 Timing buffered disk reads:  188 MB in  3.02 seconds =  62.19 MB/sec

/dev/mapper/pasmerges:
 Timing O_DIRECT cached reads:   448 MB in  2.01 seconds = 223.36 MB/sec
 Timing O_DIRECT disk reads:  316 MB in  3.00 seconds = 105.19 MB/sec

[root@merge01 ~]# hdparm -T -t /dev/mapper/pasmerges; hdparm --direct -T -t /dev/mapper/pasmerges

/dev/mapper/pasmerges:
 Timing cached reads:   9928 MB in  2.00 seconds = 4967.85 MB/sec
 Timing buffered disk reads:  188 MB in  3.02 seconds =  62.20 MB/sec

/dev/mapper/pasmerges:
 Timing O_DIRECT cached reads:   448 MB in  2.01 seconds = 223.30 MB/sec
 Timing O_DIRECT disk reads:  316 MB in  3.02 seconds = 104.76 MB/sec

If I'm reading this correctly, buffered reads are much SLOWER than non-buffered reads?  Why would that be?  Memory pressure?  Slow CPUs?

Also, the performance of the direct reads seems almost at the theoretical maximum (which I understand is about 117MB/s).

HOWEVER, when I run a simple dd of a 4GB file using direct i/o, I get this (create the file, then copy from one iSCSI device to another):

[root@merge01 ~]# time dd if=/dev/zero of=/pas/merges/perftest1 oflag=direct conv=notrunc bs=8192 count=524288
524288+0 records in
524288+0 records out
4294967296 bytes (4.3 GB) copied, 357.925 seconds, 12.0 MB/s

real    5m57.927s
user    0m0.339s
sys     0m29.942s

[root@merge01 ~]# time dd if=/pas/merges/perftest1 of=/opt/oracle/oradata/pas/perftest2 iflag=direct oflag=direct conv=notrunc bs=8192            
524288+0 records in
524288+0 records out
4294967296 bytes (4.3 GB) copied, 1233.02 seconds, 3.5 MB/s

real    20m33.108s
user    0m0.477s
sys     0m40.773s

Note that /pas/merges and /opt/oracle/oradata/pas are on two different Equallogic volumes and therefore two different iSCSI connections.

This is TERRIBLE!  What am I doing wrong here?  I just don't understand the conflicting results. :-(

Does anyone have any suggestions on what we can do to increase our performance?

Thanks in advance...

 

24 Posts

February 11th, 2010 09:00

Sorry to reply to my own post, but I just found this thread on the Powerconnect forum: 

What happened to the PowerConnect 62xx v3.0 firmware? (http://en.community.dell.com/forums/p/19294979/19654178.aspx#19654178)

They say that they pulled the 3.0.0.8 firmware due to bugs.  We were told to upgrade to it for iSCSI performance.  So, now what?  Do we downgrade to 2.2.0.3 again or sit tight?  We do notice pauses at times when accessing the SAN.  I wonder if this is the cause? :-/

24 Posts

February 11th, 2010 09:00

Thanks for the reply...

The Powerconnects are on firmware 3.0.0.8.  Funny thing is the Dell support site only shows 2.2.0.3 as the latest, but we were told to upgrade past that.

My SAN is on a 192.168.100.* subnet.  My LAN is on 192.168.10.*

I neglected to mention that the two switches are stacked, in case that matters.

The switches have VLAN 2 for iSCSI and VLAN 1 for management.  VLAN 1 is on the 192.168.10.*  and VLAN 2 is on 192.168.100.*.

I also have iSCSI Class-of-Service settings as advised by Dell Powerconnect engineers.  I can post the full configuration if that is helpful.

Anything else I can check?

9.3K Posts

February 11th, 2010 09:00

A few basics for all iSCSI (production) setups first:

- the 2 PowerConnect 6224s; are they on the latest firmware (Dell came with a firmware update for them a while ago that helped iSCSI performance supposedly)

- do you have the switches isolated (as in; not connected) from your LAN, or did you at least create a VLAN for just the iSCSI traffic?

- did you use a dedicated ip-range/subnet for the whole iSCSI network?

No Events found!

Top