Slow NDMP Backups on Netapp

Question

Hello, We are having some problems with a recently implemented NDMP Netapp backup with Networker. The backups are running really slowly, between 500kb/s to 20mb/sec at best to LTO4 drives.. I suspect this is because of the number of files on each volume, they are mail spools and web hosting volumes so they contain millions of tiny files. I have a few ideas on how to fix this, but was hoping for some advice as I am new to both NDMP and Netapp devices.. DAR backups? I've seen this mentioned in the documentation but the details are very sketchy and I'm not sure if it will help the backup speeds. I'm also concerned about what other impact it may have on the backups. Multiple streams? We get around the dense filesystem issue on our primary file server by running parallel save sets of different folders, but with NDMP we can only run a single stream. I read somewhere that we can still use an NDMP copy to the networker server which can then multiplex onto tape with 4 streams per filer head? I believe this is configured by setting the networker server as the storage node under the Filer client, and appending -M to the nsrndmp_save command.. SnapImage? Again, the documentation is really sketchy around this but from what I have gathered, this will create a snap, then rip down the raw partition at block level so it should be really fast as the file system isnt taken into account. Our Networker support provider said this: 'snapimage used ndmp, so no, you wouldn’t gain any advantage.' but I don't believe it based on what I have read so far. Unfortunately I don't have the experience to say otherwise.. Any other suggestions would be really helpful but I'd like to know what you guys think. Cheers

sweller1 · Accepted Answer

HDFS, high density file systems, have been a challenge for many years and continue to be regardless of where they reside. SnapImage was developed specifically for this but it runs on the client file system providing a host-based copy-on-write snapshot. It is this reason it will not work for the NetApp systems. DAR, Direct Access Restore, simply keeps history information to allow for faster rile location on the tape during a recovery. You certainly could use the -M option which is then leveraging the DSA, Data Service Agent, process. What happens here is the NDMP streams comes into the NetWorker Storage Node which then puts a "save" wrapper around it letting it then be treated as any other NetWorker save process. This means you get mulitplexing support, write to disk support and cloning. The device would be attached to the Storage Node now rather than the NDMP data mover so a bit of configuration needs to be done and as well it is important to understand than the traffic will now traverse your network so consideration should be made for that.

You may also consider a solution that includes NetWorker PowerSnap for NAS, Network Attached Storage v2.4. By using the existing NetWorker PowerSnap framework, PowerSnap Module for NAS Devices provides the following operational capabilities:

◆ Instant backup

◆ Live backup

◆ Instant restore

◆ Rollover of instant backup copy onto conventional media

◆ Traditional and EMC NetWorker Application Module NFS-based save set and file-by-file recoveries from the tape or disk

◆ Retention policy based recycling of snapshot resource

◆ Schedule-based snapshot creation

◆ Snapshot consistency checker (to keep the media database and the snapshot subsystem synchronized)

◆ Use of nsrsnapadmin to back up snapshots, perform rollbacks, and browse snapshots.

I am not here trying to sell you anything more, just trying to help you understand the options available to you. This is not an easy problem to solve for sure but it can be solved. We could also replace the NetApp with a Celerra and do Volume Based Backups which could saturate the device for sure.

Hope this helps a little. Feel free to have your EMC rep contact me if you need more detailed technical info on these options.

Cheers grndmstr!

-Steve

grndmstr · Answer

Thanks for your feedback Steve, really informative. I think in the short term we will go with the storage node method while we investigate the Powersnap option further. Unfortunately we cant move to Celerra just yet, we aquired a company that was a netapp shop and their gear still has maintenance left on it for a while.

On a slightly related note, we do have a celerra here at our head office (i'm still figuring out how to work it), the volume based backups sound very interesting. I'm assuming this is also done through NDMP? Do you have any references that I can read to find out more about it and how to implement? We are currently backing up some of the data on there via CIFS, so I'm sure we can do it a better way...

grndmstr · Answer

bumping this, hopefully Steve (or someone else) sees the reply in regards to the celerra volume based backups..

mfriedma · Answer

Hello,

Celerra Volume Based Backup, or in its new name NDMP Volume Backup (NVB) is based on NDMP. it provides sequential block level backup with file level restore capability.

There are a few pre-req for one to use NVB:

1. The target device must be directly connected to the NDMP server (The Celerra) and configured as NDMP device within NetWorker

2. The source file system must be in "read only" mode, the way to accomplish it is by using a SnapSure snapshot volume before the actual data transfer begins, SnapSure is copy on write based product which means that the space for the snapshot should add about 20% more capacity to your production LUN.

3. File level recovery will not work with NVB if de-dup is enabled on the source file system, volume recovery will work as spec.

In terms of NetWorker configuration, there are a very few minor changes to the client resource:

1. The backup command stays the same i.e nsrndmp_save -T tar

2. To enable NVB you should add VBB=Yes in the application information for the client

3. to enable SnapSure you should add Snapsure=yes in the application information for the client.

NW in this case, will create a snpashot of you file system (a few seconds), mount it as a read only file system on the same Data Mover, and begin the block transfer, at the end of the operation NDMP will generate and send the metadata for the files that have been backed up and release/delete the snapshot.

The performance gain, especially in your case will be tremendous compared to traditional NDMP let alone CIFS.

Hope it helps,

Mickey

kohinoor1 · Answer

Hello,

we have Networker 7.6 32 bit running on Windows 2003 32bit system. We had performance-related problems too, but tweaking the network settings in the registry solved to problem:

TCP1323Opts = 1

TCPWindowSize = 250000

The Networker client resource parameters are the following:

backup command: nsrndmp_save -M -T dump

(the -M is required for us because we are backing up to a non-NDMP device - we have a library attached with U160 scsi to the server, two lto3 drives available)

application information:

HIST=y

UPDATE=y

(required for incremental and/or differential backups)

When both drives are working and NDMP backups are running I can measure 160MB/sec data rate.

Hope this helps :>

Best regards,

Szilard Szekeres.

ddadmins · Answer

I am having trouble restoring files to my netapp filer as well.  I am getting awful speeds.  I am using windows 2008 R2 and have the same issue.  Have you tried on Windows 2008 box as of yet?  I read on technet somewhere that the the 2008 server automatically adjust the tcpwindows size according the source requesting the restore.  your thoughts?

ddadmins · Answer

have you updated your environment? are you running on windows 2008 as of yet?  I am getting terrible performance issues on my restores to my Netapp filer.  I had Netapp/EMC/DataDomain on the line and we are racking our brains.  Our non-ndmp backups locally are pretty decent but I avg 5/mb/s

Thierry101 · Answer

hi ddadmins is your 2008 server win2k8 storage server? seen awful restore speed using Data Protector last time..

NetWorker

Was this post helpful?