Very slow backup speeds using NDMP

Question

Here is my environment:
We have an EMC Celerra NAS NS502G sitting in a Colocation facility. It is hosting CIFS servers that need to be backed up. In this same colo facility we have several Windows 2003 Servers hosting files.

20 miles away we have our office where our IT staff works and this is also the location of our Network 7.22 server and backup drive.

The office is connected to the colo via a 300Mbs metronet circuit.

Before we moved any data from the Windows 2003 servers to the NAS, we were backing up data from the Windows 2k3 servers to the Networker server across the metronet at a rate of approximately 100GB/hour. Now that we are going from the NAS to the Networker server via NDMP, it is backing up at approximately 25GB/hour.

We are backing up using the following command in the Remote field on the Networker client.
"nsrndmp_save -T dump -M"
The following variables are set:
HIST=y
UPDATE=y
DIRECT=y

The NDMP backup only utilizes 1 drive on our Networker server whereas the non-NDMP backups could use multiple drives at once, although I was told that was a caveat of NDMP.

I also have a fiber attached tape drive on the Celerra for DR purposes, and if I initiate a backup from the NAS directly to the locally attached tape drive via NDMP as well, the backup speeds will increase exponentially.

Are there any tweaks that can be done for NDMP backups such as mine?

Thank you for your assistance.
Brian

ble1 · Answer

I really don't see why you can't use multiple drive - caveat is on Celerra side where you can have only 4 steams per data mover.

The speed you get is not good. While I didn't do network based Celerra backups, I did with directly attached drives using tar/dump and VBB. Usually I would get 50-70MB/s (I had one drive per data mover). So, I'm quite sure Celerra is able to perform. Since you have network in between I would check first that network card settings are in sync between both sides (and ports on switch).

briantrotter · Answer

Network ports seem to be ok. I see no errors, and they are autodetecting correctly.

I did a "server_pax server_2 -stats -v" and this is one of the things I see.
Not sure if this is helpful to anyone.

** nasa thid 3 is running backup with dump format **
Backup root directory: /root_vdm_11/DENFS1
Total bytes processed: 1737035041893
Total file processed: 819081
throughput: 7 MB/sec
average file size: 2071KB
Total nasa wait nass count: 13491
Total nasa wait nass time: 34885 msec
Total time since last reset: 233437 sec
1177355630: NDMP: 5: PaxDataModule get NULL tape device
Tape device name: unknown
dir or 0 size file processed: 70170
1 -- 8KB size file processed: 276639
8KB+1 -- 16KB size file processed: 62804
16KB+1 -- 32KB size file processed: 95295
32KB+1 -- 64KB size file processed: 68521
64KB+1 -- 1MB size file processed: 170842
1MB+1 -- 32MB size file processed: 68148
32MB+1 -- 1GB size file processed: 6387
1G more size file processed: 275
fs /root_vdm_11/DENFS1 size is: 1891772768256 Bytes
Estimated time remain is 21081 sec

ble1 · Answer

7 MB/s is not much, but I wonder if your issue is related to small files. Have you tried VBB for test?

briantrotter · Answer

I have not tried VBB. Let me do some quick reading on that as I am not very familiar with it.

ble1 · Answer

You will probably need EMC support (Celerra) on this as VBB requires a file (actually a file system) on each DM which is used by restore operation of that kind. Once Celerra is configured for VBB you can start enjoying block based backups which should make difference in performance and speed you see.

briantrotter · Answer

I did speak with a Celerra engineer and he swayed me away from using VBB as it is doing volume backups, and the majority of our restore requests are file level restores. So this probably wouldnt be the best solution for us.

While doing some testing, I found that each backup thread from the NAS backs up at around 7Mb/s. So if I have a backup session going from the NAS, it is backing up at around 24Gb/hour. However, if I start another backup from the NAS, it too will run at the same speed. I almost expected to see both jobs running at 12Gb/hour to total 24Gb/hour, but it didnt. It seemed as though each job was backing up at the same speed.

That just seems to confirm there is some type of configuration somewhere that is limiting each backup thread to only go a certain speed.

ble1 · Answer

With VBB you can do file level restores.

ajw2 · Answer

You can also do file level restores without VBB.

ble1 · Answer

You can, but your backup will be slower with large amount of smaller files - check pax stats above by Brian.

NetWorker

Very slow backup speeds using NDMP

Was this post helpful?