How big is too big?

We are backing up our main file server traditionally.  It consists of two volumes for home and shared folders.

Home 3401 GB and 3963714 files

Shared 3931 GB and 3929367 files

The last full took 43 hours 39 minutes.  The volumes are part of a physical cluster, one HP DL380 G6 and one DL360 G6.  These servers are not Storage Nodes-- just regular backup clients.  They have 1Gb network connections.

The volumes are hosted on a CX4-240 (4GB FC).  We back up to a DD880 using Networker 7.6.2.2 Build 651.  The DD880 and Networker server are direct connected via 10GbE.

Let me know if you need any more info.

David

DDBOOST.  Please don't remind me of VTL!

Thanks for the info sounds like you are getting better performance than we are.  We are using almost all the same hardware as you are.  Our virtual enviroment has a dual 10GbE connection to the core, our DD670 is connected to the core with 6 1GbE and was originally setup as a CIFS target for NW.  We have now changed it over to DDboost.  How are you using your DD880, (DDboost, CIFS, VTL) ?

I even made the server its own storage node and it only backs up itsself.

You mean just backing up C:?

and when you tried VADP was it using nbd or san?

This all seems a little confused, I think we need to define the problem a little more clearly.

Firstly you state you think it might solve it by using Avamar, but you are already using DD Boost so you are already using deduped backups so there is no reason to assume that would solve your problem.

Secondly 2TB isn't particularly big (and it appears you only have 1.5TB of data anyway), we have a 7TB SAP backup that takes 3 hours to backup; EMC's documentation suggests you can get up to 26.3TB an hour using DDBoost over a 10Gb network.  You seem to be getting 0.04TB an hour, that is the equivalent speed of an LTO1 drive.

Thirdly, you state this is a "traditional" backup, but you also state you are thinking of moving your NAS head, that it is a virtual server you need to do VADP and file level recovery with.  My assumption from what I have read is (1) this is a virtual server (2) backing-up via a proxy (3) you are backing up over 4GB FC.

These backups seem to be going significantly slower than you would expect; you need to analyze each part of the chain and ascertain where the bottleneck lies.  Check if there are issues reading the data from disk, that the proxy is not being overloaded during the backup, that the data is correctly being sent over the SAN etc.

Thanks David,

Your right my main issue is that performance seems to be lacking.  We have engaged tech support several times over the past 2 years and they assure me that this is normal performance.  We also engaged EMC profesional services and they came out and accessed our enviroment and really found nothing.  I have monitored the servers involved and they do not appear to be overloaded.  The network is not impacted.  So far I havent found anything to point too.

The server in question is a VM.  If I do a proxy based VADP image level backup the performance is pretty good but it still takes 3-4 hours.  If I do a proxy based VADP image w/file level recovery capabilities it takes 3+ days, it takes forever to build the meta file.  Currently I have made the server a storage node and it backs its self up straight to the DD via traditional client based backup.  This backup one weekend will take 40 hours and the next time take 80 hours for no good reason.

We have also enabled ddboost and for the backup I have not seen a performance increase at all.  The clone between the DD's has skyrocketed, it went from 60-70Mbs to over 600Mbs which is impressive.

So far EMC's answer has been to use Avamar, or move to a NAS head in the VNX and use NDMP data movers for backup. 

Idealy I would like to have a 10Gb network dedicated to backup, but I cant even get it to perform very well on a 1Gb.  I have tried to find better info from EMC on what kind of speed's I should expect but have yet to find anything.  When I ask them directly I never seem to get a straight answer. 

So I came here to get some info and see if anyone else is having success with something similar or not.  As I stated before we are looking at implementing home folders and that could easily be well over 10TB and I have no idea how we would ever get that backed up with Networker. 

Seeing that you are having good success backing up 7TB in a resasonable amount of time does give me some hope.

thanks, John.

From your answer John, it appears the point where everything breaks down is in doing the file level recovery which makes this a VMWare issue primarily, it may be worth engaging their technical support to investigate this issue further.  I am interpretting your workaround as installing the client within the VM and backing up from there which takes 40 hours as opposed to 3+ days with file level VADP recovery.  I don't have any immediate answer why this would occur but I'm curious as to what kind of files you have on this system.

If your cloning speed has increased radically it suggests to me the data is being deduplicated, just that the slow response of the client is not making it evident in terms of faster backups (your network is not being pushed to the limit beforehand so deduplication does not push backup speeds up).  There does not appear to be any clear logic as to why Avamar would resolve this problem, Avamar will transfer the deduplication processing to the client where we have already identified where one problem lies.

This file server is for our admin staff, there are over 3.8 million files.  Unfortunatlely we have no policy on archiving or anything so everyone just keeps anything and everything for however long they want too.  We have the same problem with email, Exchange is almost 3TB with about a 200 gig a year growth.  No one will make a command decision that might be political suicide.  Change at a school district is painfully slow.  Still I have to come up with a way to back up all this data with what resources I already have.  The actual backup with VADP is pretty quick it is just the time it takes to create the index meta data for the files.  Again EMC tech support said this is pretty normal and what I should expect.

I think the Avamar thing got tossed around by EMC because its what they are pushing now and they want you to buy it.  To me it doesn't look like ddboost is working like it should on the actual backup ie when this client or actually any client does the initial backup.  Once again EMC tech support said this is normal and was suprised by the speed of the clone.  So far tech support has not instilled a whole lot of confidence in me.

What it comes down to is are the speed's I am seeing normal for client based, VADP, and VADP /w file level recovery backup "normal"?  If so then what other options do I have to backup large volumes of data?  The VADP is decent for backup but file recovery takes all day at best.

BTW I am no backup expert.  I became the backup admin by default so everything I know comes from my experience of working the Networker for the past 2.5 years.  I have become fairly profecient with Networker and the week long class helped fill in alot of the blanks but there still seems to be alot of other ways to skin the cat out there other than what I have implemented with Networker.   

EMC's tech support is variable, you tend to find it improves if you give your account manager a hard time...  I still think the VM environment is the cause of the problems rather than the backup environment, though it also could be down to the large number of files on this server, each file needs to generate meta data to track what is backed up which seems to be what EMC's tech support think.  Have you thought about backing up data in smaller chunks or even moving this server into smaller VMs?  Also what is the spec of your proxy server?

One important thing to make your management aware of, the time for a backup is a reflection of minimum time for a recovery.  If they have a server failure are they happy for a potential 2-3 days downtime, perhaps longer, are they happy with that sort of RTO?

If I had homes and profiles on SAN/NAS storage, I would use storage snapshot technology to create backups.