AX4-5i bottleneck in disk test

First, I have yet to meet anyone that is a SAN expert.   We all have stuff to learn.  In regard to your question, the backend BUS is 4GB.  You could be dealing with a disk issue (wrong disk make-up for the test) or a SP Cache issue.  Check the advanced perfoance graphs in analyser and see what the dirty pages look like on the SPs.  If you see 60-80% that is generally normal (configurable on the SAN -- but don't change it).  If you see 90+ and then fast drops to near 0-30% then you are probably experincing Forced Flushing (which in that case I would guess that the test your are running is not 10% write).  I suspect the real issue is the iSCSI going over the network, though without data hard to prove. Then again, I don't use iSCSI so I have really don't know what the realistic expectations are and if I tried to guess I am sure one of my peers here would smite me.   Other than that, to begin to look elsewhere, we would need to know what the test are, what configs you are pushing them to, and/or much more info.

On a side note, I would expect, at the disk level around for that usage pattern:

3 Disk R5 @ 15K = 28-32Mbps

8 Disk R10 @ 15K = 500-550Mbps

Hope it helps.

One minor correction, the back end bus on the AX4-5 is 3Gb/s SAS, I suspect you were confused with CX4 which is 4Gb/s FC.

The IO profile you are testing is very unusual, does your application actually use a 2MB size for Random reads and writes?  Typically the largest we see is 512KB or so and that is generally more sequential and in data warehousing applications.  Anything 2048 blocks will bypass read and write cache, which will make the performance entirely disk dependant.  Also you will want to be aware of the concurrency of the workload, how many threads are running in this test?

With IO sizes that large, you are almost certainly getting stripe crossings for every single one, which is going to have the drives working  very hard. You may want to consider using metaLUNs for this test/application so that the stripe element size can be made very large.  Check the best practices document for more details:

http://powerlink.emc.com/km/live1/en_US/Offering_Technical/White_Paper/h5773-clariion-best-practices-performance-availability-wp.pdf

Thanks for the reply.

When you say to check the performance analyzer, do you mean the Symmetrix Performance Analyzer, or is there another program for checking performance? I've been looking through the Powerlink site, and that seems to be the only thing I can find for analyzing and I'm not sure that we have a license for it.  Currently the only thing I've used to mess with the SAN is the Navisphere web page management.  Pretty big newbie when it comes to all of this .  Are there any other programs that are must haves for using with the SAN?  Powerlink site isn't really the best for getting around.

Full specs of the system:

2 - HP DL360 G7 servers running Vmware ESXi 4.1.  Ran the test from a VM on one host and got 46mbps.  Ran it from VMs on seperate hosts at same time and got 23 on both.  Which would seem to rule out the problem being on the servers side.

Servers are connected through iSCSI to a 3 switch group of Cisco C3750 (48 ports each), which is the backbone for the company.

SAN is EMC AX4-5i with 2 additional shelves.  33 drives total, all 450gb 15k SAS.

Disk test tool is passmark advanced disk test http://www.passmark.com/products/pt_advdisk.htm

The software tool has 4 built in tests to simulate different work loads.  File Share server, Web Server, Workstation, and Database.

Database test is File size: 2048 MB, Block Size 2048 Bytes, Standard Win32 API (uncached), Asynchronous request mode, IO Queue Length 128, 90% random, 10% sequential, 90% read, 10% write.

File server test is File size: 512 MB, Block Size 16384 Bytes, Standard Win32 API (uncached), Asynchronous request mode, IO Queue Length 128, 100% random, 0% sequential, 80% read, 20% write.

Web server test is File size: 4095 MB, Block Size 16384 Bytes, Standard Win32 API (uncached), Asynchronous request mode, IO Queue Length 64, 100% random, 0% sequential, 100% read, 0% write.

Workstation test is File size: 4095 MB, Block Size 16384 Bytes, Standard Win32 API (uncached), Synchronous request mode, IO Queue Length 0, 80% random, 20% sequential, 70% read, 30% write.

result numbers are mbps averages of read/write

4 disk pool with Raid 5:

File share: 30.9

Web Server: 43.1

Wstation: 4.7

DB: 46

4 disk pool with Raid 10:

File share: 30.1

Web Server: 42.4

Wstation: 4.8

DB: 46

8 disk pool with Raid 10:

File share: 52.1

Web Server: 63

Wstation: 4.9

DB: 46

Eventually I want to make a single 20 disk raid 10 pool and put all virtual servers/file shares/etc onto it.  I've been recording differences between raid types and disk pool sizes to make sure this is the correct path, and was a little perplexed when I added another 4 disks and got no change on the Workstation and Database tests.

Ok, I installed the Navisphere Off-Array management UI thing and just enabled Data logging.  Hopefully I can Analyze some stuff on that after its collected some data.  Still waiting for my disk pool to finish expanding from 8 to 12 disks, so I won't be able to do some real testing till tomorrow.

You didn't mention what operating system you're using, but you might want to look at this Knowledgebacse article on PowerLink emc150702 - this is mostly for Read IO and your test sounds like mostly reads - this is for Windows, emc264171 for ESX.

glen

I'm running the test from a Server 2008r2 guest on a VMware ESXi host.

I started looking into my network and saw the % usage showed 100% on the switch for one of the two iscsi ports, while the other was only being used for failover.  So after a ton of reading on VMware and EMC, I finally got round robin going accross the two iscsi server ports connected to all 4 paths on the SAN (2 ports per SP).  Since its an Active/Passive array, I'm guessing 2gigabit to only one of the Storage Processors is the best I can get?  Looked at the usage, and I'm seeing a nice pretty 50% usage accross all ports.  Sadly, tests are running at the same speed as before so it looks like that wasn't the problem.  Has anyone done any test for how many bytes to send down each path at a time?

http://curbstuff.com/mypics/SPA_2011-09-08.naz

http://curbstuff.com/mypics/SPB_2011-09-08.naz

Here is some data logging I did.  I simply started the data logging, ran a bunch of hardware tests in Passmark and HD Tune, then stopped it.  Should give a pretty good look at what is going on, so if someone who can view these files could take a look I'd be really grateful.

I recently found a thread on another forum with a guy running the same AX4-5i SAN, with a 14 disk r10 7200rpm SATA disk pool, and his results were blowing my 12 disk r10 15krpm SAS disk pool out of the water in HD Tune Pro.  Some of my tests were a little faster, but 90% of them were all MUCH lower.  So right now I'm pretty much positive that I've got something seriously wrong.  I would think my disk pool would absolutely destroy his.

Your Exchange LUN (2) is driving about 2000-4000 IOPS for most of the test and then drives over 20,000 IOPS near the end of the test. A 12 disk R10 group should do about 1800-2000 IOPS in an OLTP, low latency workload. If you were to run the test on two LUNs simultaneously, making sure that each LUN was owned by a different SP, you’d probably see better performance. In your test, SPA is running close to 100% CPU Utilization. The 20,000 IOPS at the end is a result of 100% read cache hits.

A single LUN typically can’t drive the maximum IOPS from a RAID Group, you usually need 2 to 4 LUNs.

Richard J Anderson

Hi Ted,

How did you get this 500-550MB? 8 disk R10 are you considering total 8 disk(4+4) or  16 disks(8+8)? Even with 16 disks what is the calculation behind?

Ok, this is really putting off my big project to move the organizations data to a new drive set.

Right now Glen has been working with me, and he has found that some of the windows 2003 servers are "constantly trying to log into both SPA port 0 and SPB port 0 - because there are already logged in or a host with the same IQN is logged in, these are being rejected."  Problem doesn't seem to be happening on the VMware servers, which I had setup.  Win2k3 servers were setup before I got to the company.  Problem seems to be happening whether the servers even have drives presented to them or not.

He also recommended running the "naviseccli -h 192.168.1.29 -user admin -password ******** -scope 0 port -removeHBA -host hostname" command to remove inactive connections.  The command seemed to run fine, but Inactive connections are still shown.  I have Navi express, and it still shows inactive connections.  Not sure if this matters or is causing the performance, or if it is just not very tidy to have.

Any help is appreciated.