Performance tuning for NDMP backups

Question

This should be a fun one

.

I have CFS-14 connected to CDL 720 via FC switch, 2G. It's my perception that my backups are running slow and could be tweaked to do a lot better than ~30MB/s. For example if i have 4 NDMP backups running at the same time i am getting around 30MB/s per backup..so a total of 120MB/s, i would expect a little better more since we are connected to 2G interface and could possibly get close to maybe 180MB/s ? But let's say 120MB/s is all my Datamover/CDL can handle. So i have 4 concurrent backups running at 30MB/s each, now 2 backups finish and now there are only 2 backups running, how come the speed of these backups does not go up to 60MB/s each ? The remaining two backups continue at 30MB/s even though i know there is more room for throughput. I have run server_pax and went through the flow chart on page 55 of "Configuring NDMP Backups on Celerra" and i can see that in my "NASW STATS" section all buffers are in Get Pool. According to the flow chart it means that my tape drive may be the bottleneck, maybe i would buy that if this was a stand-alone tape drive, but it's a CDL. Is it possible that virtual drive instance on CDL can only handle 30MB/s ? I have set my drive buffer size to 384KB (IBM 3592 drive emulation), all backups are utilizing SnapSure. I have monitored CPU/Memory utilization during backup and those are not the problem. (cpu idle_% = 90 and memory free(kB) = 2391619)

Your suggestions are greatly appreciated.

Thanks

urle · Answer

While playing with ndmp and IBM TSM/Tivoli I've tested several different connection methods. Direct connected tape (FC 2GBit) on my CFS-14 with 510 movers and 5.3 got 40MB/s max to an IBM Magstar 3590 single stream per mover. My NSX with 5.5 and iSCSI -GBit gets 75MB/s out of filesystems with few large files and down to 5MB/s in the student-filesystems (zillions of small cookie-sized files and roaming profiles) writing through the tsm-node into a storage-pool or to tape . I'm always bound to the filesystem and not to tape or storage-pool.

Did you try PAX-parameter writeToTape=0 to eliminate the tape as bottleneck just to see what mover can stream?

dynamox · Answer

we decided to by-pass NDMP and used server_archive to write directly to tape. From our observations we are still seeing only around 35MB/s throughput, backing up large files. Next test was to eliminate Celerra from the picture, so we used tar on an AIX box and wrote directly to tape on CDL ..same result ~35MB/s. We are suspecting CDL can not handle more than ~40MB/s per stream. If my backend on CDL is Raid5 (4+1), 500G SATA II drives, what throughput should i expect from one raid group in a sequential write ?

dynamox · Answer

for those who care

to see if the file system on Celerra was capable of providing data at more then 35MB/s, i created 4 virtual mount points within TSM and started NDMP backup on all 4 virtual mount points. So i have 4 virtual tape drives on CDL backing up one file system on Celerra and i am getting around 20MB/s per NDMP stream for a total of 80MB/s. This tells me that my Celerra is capable of doing at least 80MB/s (thanks to AVM for spreading the load :)), but my single virtual drive on CDL can only handle 20-30MB/s. I should be able to get more out of a R5(7+1) raid group on CDL (750G SATA II drives) ?

Celerra

Performance tuning for NDMP backups

Was this post helpful?