bad performnace when several AIX 5.3 servers are backuped in the same time

I would suggest to run client initiated backup from one AIX machine and then run second one while first one is running. During that period have sniffer running on both AIX machines and backup server/storage node and see what happens. Also expand that test with some things like having one AIX box writing to one tape and another to another (you wish to see if issue is there). Performance problems are fun and challenge, but they require some tests to be done first. Are these boxes multihomed? I find strange that you get max of 100Mbit network for backup of single box and ftp runs over that (may indicated multiple NIC and perhaps mismatch in mode for the one used for backup traffic).

Hi,

The 4 AIX 5.3 are on two bull PL450 and each server have tow physical ethernet card on for the production and an other (disable for the moment) for a futhuer backup LAN. the ethernet are in Gigabyte full duplex.
The networker server have 2 ethernet card one for the production and backup and an other for an other backup network (for skip firewall).

We try some test with save command.
4 save simultanous started one after the other the first and second begin normaly, but after the third and fourth, the backup run very slow 245kb.. with up to 15mb. we killed the fourth save and the performance grow to 12mb constantly with less bad flow to some Kb.
We have restart juste 2 save to see if the performance was right, in fact 2 save unde two servers, we backuped the same that in the first test (Oracle binarie 4gb) and the average of the backup was 22mb 8gb in 9minutes so good performance !!

So when we run 4 save from 4 servers, the performace are very bad but when we run 2 same save on 2 same server the performance are right !!

We have run a last tests 4 save from juste 1 server. the backup begin correclty 15mb / 35 mb... and during the babkup, during 2 minutes it run at 340kb, 240kb...and some time 12mb 20mb, 2000kb... and 234kb... so the same problem
We have run "top" to see system resources on tha AIX 5.3 it was 86% idle !! the networker server is at 95% idle

In fact i thinks that the networker server can't manage more than 2 or 3 flows into simultaneous from one or several servers.

Have you go an idea...
Many thanks for your help

Best regards

Jean-Charles

Can you do following - do 30GB big file (random data if possible) and place it on all 4 servers.

Run save in parallel from each node and if you get same error perfect.

Then repeat this test by using scp bigfile root@solaris_bck_srv-bck:/dev/rmt/....

What is important here is following - make sure that with scp test you use same devices in same order as with save test (I must say that I assume scp will write to tape, however make sure you load some dummy tapes or those you can destroy and make sure they are rewinded). Second important thing is to initiate scp towards backup interface name of server (thus local routing will take care that local bck NIC talks to remote bck NIC).

It would be nice if you could use snoop to monitor packages on Solaris box. If you get the same then we know it is not NW. If you get it different, time to open SR.

Just a comment, I have seen a similar problem with my AIX servers backing up to a NetBackup server. Since the environments are different, this info may not apply.

One change we made that improved the problem was, made sure all the tape traffic used one hba and the disk traffic used the other. Since sending tape and disk traffic over the same HBA is not a best practice. It is likely the above does not apply to your env. -Tim