FA Utilization

Complex question !!

1) number of IOps depends on IO size. The larger the IO, the less IOps. The larger the IOps, the higher the throughput (Mb/sec).

2) size depends on OS/Application .. The host is the initiator, thus sending commands (IO) to the storage that can simply answer

3) each DMX FA processor serves 2 different ports. Each port have its own logic that handles low level details of communication, however both ports share same processor (and same logic). So you have metrics regarding each port and metrics regarding the whole processor. AFAIK low level logic limits number of iops, processor limits througput .. however you may receive better/different opinion on my last sentence.

4) I'm sorry .. no knowledge of how a CX works

Dear srichev,
I've moved this thread also from the Powerlink Lite users forum into the Symmetrix one since you are now registered as a customer and have full access.

Julie Gibson
Forums Admin

Have Aran and Stefano answered your questions?

What tool are you using ?? WLA/performance manager or symstat or whatever ??

EMC does not publish exact performance numbers so you will not get an exact answer for the number of iops an adapter can handle.

Think of it this way:

The FA port shows your throughput in MB/sec which is a function of your link speed. If you are connected at 4Gb for that specific port then you can push (in theory) up to 500MB/sec down that port of raw data. Minus some throughput for network overhead and reality of course. So in theory if you have an 8-port card with all ports connected at 4Gb then you have 4 GB/sec of bandwidth.

The FA (Dir-Fibre in WLA) shows the CPU associated with port 0, 1 on a card. This has a direct limitation on the number of I/O per second you can perform down BOTH ports aggregated. I have not found nor seen specifications for the DMX-3 & DMX-4 - many years ago I went to a performance class when the DMX2000 was the best thing and was told the limit on each FA processor was about 2,200 iops.

Looking at data on a DMX-4 I see about 4000 iops peak per FA port, but utilization is under 30% Making a swag I'd say maybe each FA processor can handle 6k - 8k of I/O before performance degrades. This is a guess and it is unlikely you will get an official answer from EMC on this

In short - FA utilization is the sum of port 0, 1 for a given CPU (A, B, C, D) not for the entire card. If you have an FA utilization that is higher than your combined port utilization totals then you most likely have very small I/O blocks which are putting additional strain on the CPU to process.

I don't think it would be possible (except maybe under perfect test cases) to ever push the bandwidth on an 8-port fully connected FA to its limits with the FA processors becoming a limit way before the ports themselves.

That said - I have yet to see a case where the FA ports have been a limit - usually you don't have enough drives on the back-end to keep up with an FA port until maybe you start playing with flash drives

Stefnao,

Thank you for the response but not clear about the Processor part.In a DMX-4 FAs have 8-ports.Assume 2 ports share the same processor(mean 0,1).Then we will be having 4 processors.While i was looking at the stats per port utilization is 3-5% and the FA utilization is 23-35%.Is the FA utilization is the sum of all the 8 ports?

I tend to agree with what you said



It depends on link speed .. iops is bound by link speed .. faster links means more iops, obviously .. How fast were the ports you was playing with ?? I guess 2 gbit.



Again .. I suspect you are right .. However in EMC we have so called "speed guru" that are the only guys allowed to talk about actual numbers. An "average" EMC employee can't discuss numbers with customers. A speed guru can



Nice point .. However I have different examples, in case you need them
Somewhere we had to add FA ports, thus allowing more iops against the same devices.

That 2,200 iops number was given to me in a class and it did assume 2Gb link speed if I recall correctly. I don't recall if we ever achieved that number in real life. The other numbers were just some observations from data I see on our production DMX-4 and a quick educated (sic) guess on potential limits.

The way I understand it iops are bound by both CPU and link speed - small iops mean you have a better chance of running out of CPU horse power to saturate the link while large I/Os mean you could possibly run out of link bandwidth. Same concept as network throughput really - something has to fill that link with data. Your CPU throughput isn't tied directly to your link speed, but it certainly can be insufficient to drive the link to full speed. I strongly suspect the FA processors are not fast enough to fill 8Gbs of bandwidth using 2kb chunks of data

I guess I've been lucky though just to be clear I'm not implying (1) FA/HBA connection is adequate for all or event most applications. Have had numerous setups where I/O intensive servers have had 4 or more FA connections (largest I've setup was 12 for a large warehouse environment) to drive throughput beyond the normal 2 used for simply redundancy.

Wow ...12 dedicated FAs for one host ? How many HBAs did it have ?

This is a large Data Warehouse box - the mapping is one-to-one so we have 12 ports though I believe it is really 6 physical HBA cards with dual port connections.

The host can drive these cards pretty good at times - easily the most I/O intensive application we have.

Uhm.... 4Gb is actually 400MB because of the 10/8 bit conversion the Fibre Channel protocol uses So to get 4GBps you need 10 ports, not 8.

CPU = processor? I think we have a translation problem since I think these mean the same thing

The shared CPU bounds the number of iops down each FA port that share that CPU and the actual optics/controller for the ports limit throughput. As the thread you reference states the "hardware plumbing" limits throughput not the CPU.

Some of this is semantics, but the FA "processor" you see referenced everywhere is the CPU that drives 2 ports on an FA card. The "plumbing" controlling the actual transfer with each port is not called an FA processor.

I suspect it's easier then what you think .. it's simply a typo that I'll fix soon