DMX-4 hypers seem limited to 3MB/s, no cache utilization?

Hi mkirsch,

Can you please tell us in which type of disks these logs are located? FC, EFD, SATA ?

The meta's are striped or concatenated ?

Do you have SRDF/S enabled on these devices ?

Kind Regards,

Kleanthis

300GB FC disks

2-way mirror hypers

striped metas

no SRDF on the LUNs I'm troubleshooting now

How would I do that? The only way I know is to hire EMC Professional Services to do a performance analysis on the array, and I don't have $5000 lying around.

Besides, this is not limited to just this 14-member meta. There are 242 separate hypers assigned to this system and they all display the same behavior... We're talking about 484 spindles. Literally days of work if I had the foggiest idea how to go about it.

Generally redo log writes should be sequential and go directly to cache and the backend should have no impact at all.  So I suspect the IO size is small and you are reaching the front-end limit for IOPs in this case.

You could try a simple IOMETER or IORATE test on a device on the same port just to verify the hardware.

That is one possibility that's on my list, but I just have no idea how to prove it.

I don't know this IOMETER or IORATE test you speak of. Is it a utility that will run on a server, or something that must be run on the array? Google is not helpful on this.

By "same port" do you mean same FA port on the array, port on the switch, or port on the server?

Here is the link to IOMeter

http://www.iometer.org/

It takes a bit of getting used to, but it is a commonly used utility for performance testing.

Both are IO driver tools.  IOMeter is for Windows and IORATE for Unix hosts.

If you could use as much of the same hardware for the test, that would be best, same LUN perfect, if you can switch your log devices.

Or just find out the IO size or IO rate on your FA CPUs and I can tell you if they are close to the maximum throughput.

Is the write response time OK?  If so, your MB/sec rate may not be an issue at all.

Hi MKirsch,

how many paths are you using for that Linux host?

Did you try to migrate the REDO Logs to a larger meta device, let's say to 8-way-meta?

Maybe if the issue caused by the Redo logs devices that affects and the data files devices and in general the normal operations of the host.

Also if the production devices have R2 devices then the SRDF will agrravate every device with at 1 ms to 2 ms more.

Have you tried to use PP?

If you are using th Native MultiPathing of Linux is that software correctly customized?

Low write hits could be an indication that there isn't any write cache available.

Has anyone from EMC looked at any performance data?

Writes should go directly to cache on Symmetrix, unless the backend can't keep up with the rate you are filling cache.

Write throughput is poor too. TOTAL read + write is only 3MB/s per hyper, only 2-3% hit rate on cache for writes too.

The metadevice is already 14-way, striped.

Using Linux native multipathing. How do you determine if it is "correctly customized?"

Hi Mkirsch,

when i mean correctly customized, i mean if you have set multipath.conf file correct with all the EMC devices inside, if on that host is installed only one Multipathing Software, because i have faced sometimes on some hosts, customers have installed PP and NMP together.

I am trying to make troubleshooting MKirsch in order to help you.

The fact that you have only 3MB/s throughput is not low only, is extremely low.

As Quincy mentioned everything must be written in the cache first and then data will be destaged to disks.

And the read must be done on the cache.

But try to start the troubleshooting the easy way first.

Is it possible to collect BTP files?

Have you Performance Analyzer or do you have SPA?

Also from SPA you can collect BTP files.

Then try to see the %utilization on the FEs that the devices resides on.

Then the %utilization of the BE that those spindles resides on.

Then as Kleanthis said try to check the utilization of the spindles that the meta's are on!!!!!

Do you have queueing on the FEs or is there any queueing on the devices?

How many disks are on the specific disk group?

And in how many BEs those are spread?

have you checked the WRITE MISS% and the WRITE HIT% for those devices?

Redo logs must resides on FC disks.

Maybe it is not host issue and maybe for the specific application you muts create a DCP and keep all those devices under that DCP in order to increase the device's performance and the throughput as well.

I used to have a thing in my ECC console called "Performance Analyzer" but it disappeared when I upgraded from 6.0 to 6.1 and I have not been able to figure out how to put it back.

I tried to install SPA last year, but I could not make it work and I couldn't get any help with it, so I gave up.

I think I know what you mean by BTP files. It's been a while since anyone has asked for them. I certainly have no idea what to do with them.

I have no way to check utilization on spindles, even if there was enough time in a year to do that for 484 spindles! It's not just this meta that's exhibiting the problem, but ALL the devices allocated to this system. 3MB/s is it, system wide.

This is the only server that pushes the array at all. If you watch the switch activity for the FA's they are all nearly idle almost all the time, except for 7A0 and 10A0 which are connected to the backup servers for BCVs, and then only for a few hours in the early mornings.

Last time EMC did a HEAT map on the array to try to sell us a new VMAX, it was very good. Only a couple of "hot spindles" but they wouldn't tell me which ones because we didn't pay for the analysis.

I was able to get someone from live chat to check the write pending status in the cache, and he said it was extremely low, only 9%. Still only 3MB/s per hyper.

This whole performance problem thing is totally new to me. I've been administering these things for 5 years and perofmance has always been awesome. Never had a problem that couldn't be attributed to a hardware failure.