The Future of NetWorker

Hi,

These are a few of the things I'd like to see in upcoming releases.

Most important is stability and robustness. Improve performance of core NetWorker (and also NMC). When it comes to features:

Logs from all restores centrally on NetWorker server for audit purposes, not scattered and overwritten locally on clients only.

Improve manual backups, ie develop a queue system, and not wait for a slot by trying every 30s-5 min randomly.

Improve nsrjobd and its db. If there is a problem with the db (and/or /nsr/tmp files), NetWorker should be able to remove/rename those files itself instead of hanging. It would be even better if it never became corrupted though.

Enhance security model so NetWorker fits better in a hosting environment. Enhance security overall.

Nsrck checks to be done in parallel for faster index checks and start-up of NetWorker.

Thanks

Rickard

Hi.

I think Rickard brought up some very good points, and I like to emphasize on the security model in NetWorker which is flawed (Oracle and NMM clients need to be admins on the entire datazone = incredible).

Also I want to add: It would be very good if it was possible to exclude VSS savesets such as VSS SYSTEM FILES, VSS USER DATA and such, because you might want to backup those savesets differently. Today the only workaround for that is to use include lists of all the disks and folders you want to backup. That does not work in big datazones. E.g. if you have Avamar integraded with NetWorker, you get very good dedup ratio for VSS save sets. But conventional NetWorker backup method needs to backup those save sets full and that's alot of data every day. I want to backup the VSS save sets with Avamar, and exclude from the normal Filesystem backup with exclude list.

Please Mr. NetWorker start fixing some of those obvious flaws in NetWorker.

Johannes

This is exactly what I was asking for - direct and honest feedback.  I have asked the Director of Product Management to make sure what is being communicated here is read and understood by the team as we factor the NetWorker roadmap.

These all sound like good improvements for Networker in the coming future, something I would like to see is a redesign around the Groups layout, similar to the NMC, where you have a top level folder lets say Windows Server Groups and under that would be all your groups related to those servers, another for UNIX, and Exchange, etc. I have 40 or so groups and can get quite cumbersome at times.

Just a thought...

A few features that are available in competing software packages which I'd like to see in future releases would include..

Encryption key management

Outage windows or at least the ability to stop and start all backup schedules without having to change anything

Bandwidth throttling by client and group level

Ability to pause/resume backups

Ability to split out savesets into multiple streams

Better MS Exchange and Sharepoint granular brick level backup/recovery

Regards,

Jeff Morton

I second Jeff Morton's list with these additions:

• The ability to stop a job using the NMC interface. There are times when an automated mount request hangs around for hours if not days.
• The ability to start the NetWorker services in a "single user mode" so we can do maintenance without every service running.
• I haven't used the new NMDA module but the NMC and the NMI module don't provide us with accurate % complete status. The same goes for NDMP backups.
• Cloning. NMC doesn't provide accurate source volume, source device, source pool, duration, % complete, size and total size info.

Bob Schuknecht

I think the most import task for NetWorker is to timeline its development with other vendor release dates.

i.e. When Microsoft or other vendor's release new versions of an Operating system or Application/Database server, NetWorker supports it within a maximum of three months from release date.

Other vendors such as Symantec (NetBackup) are acheiving this already. Windows 2008R2 and Exchange 2010 were supported "on" Microsoft's release date.

Hello!

The larger a data zone becomes, the more we think about standing a second one. One deterrent to doing that is that one has to remember which clients go to datazone 1 and which to datazone 2. The 2 have to be manually balanced. In a dynamic environment, users restores their files daily and need to know what Networker server backed up their data.

At EMC world 2011,  I asked if the concept of clustered datazones was on the roadmap, where:

- you dont care what Networker server backed up your data, as long as you can restore it.

- You add clients and they are defined based on capacity and priority.

I was surprised to hear that it was not, not even remotely.

What are the roadblocks may I ask?    

Julian

Hi Julian,

I did some checking on this for you, here's what I found.  This is on the radar from an architecture perspective.  For work flows like this to happen though, It does require some "plumbing" on the back-end. 

I hope this helps.

Allan

Thank you. I hope zone clustering gets on te roadmap soon.If the only roadblock is some "re-replumbing", we may see it soon.

Hi Julian - The concept of 'federated data zones' has been in the idea hopper for a while.

What caught my attention was the problem description...larger data zones, starting a second, etc. Federated Data Zones is one way to address that need. But if the data zone itself were more scalable it could could sustain a larger number of clients, devices, jobs, etc. while maintaining performance and minimizing management points. This is the approach we will take in 2012 with the next NetWorker release.

Best regards,

NetWorkerPM

One datazone with enough storage nodes can go a long way. The toughest thing about 1 single datazone is  not necessarily the performance. It is the size of the client indices. It continues to grow and we continue throwing at it the largest available internal drive for the platform.That is not sustainable.

Thank you for your elaborate analysis. See my comments below:

"Nowadays you rarely see people keeping critical data on local disks - so this might not be strong business case"

This is critical Networker /nsr/index data and needs to remain on local disk unless SAN storage provides better. You are right, huge index size does not make a business case for re-engineering Networker.

"How different is this from having two separate servers?  Both will have the same problem - data might be a little bit here and little bit there so how do I restore my backup cycle then? "

Having 2 data zones provides the flexibility of spreading clients over the 2 zones. Each client will eithe be backed up either by zone A or zone B. The advantage is when you want to recover, you do not have to know what zone backed it up.

"What about distributing database across server and storage nodes?"

I am looking at how I could do that.

Nowadays you rarely see people keeping critical data on local disks - so this might not be strong business case.

If you have issue with index db size, you can split it across several disks - keeping everything on one disk is case of large index databases is not best practice.

While I do not have issue with index database (as I use retention of 1 week - after all, it is just backup), I'm aware that large volumes might and will cause an issue performance wise (think browsing).

When I think of what you propossed, I see several problems.  You say you do not care which server backed up client as long as you can restore it.  OK, imagine that you can do that.  How different is this from having two separate servers?  Both will have the same problem - data might be a little bit here and little bit there so how do I restore my backup cycle then?  OK, you probably thought of that and meant to say that both servers would be aware of all backups so you would browse one view of combined data from two servers OR data between two would be synced somehow (mater-slave relationship in active-active mode) and you start the restore and voila.  Well, I doubt this would not cost performance and I really do not see the gain.  If you need more space for index, just add it.  Disk is cheap nowadays.  What about alternatives?  What about distributing database across server and storage nodes?  Well, I remember this during my NetBackup days, at least for media database.  mdb is not big and does not need distrubution (though replication of bootstrap, which is media database and resource database, across server and storage nodes is welcome IMHO).  Distributing index across servers - again, not sure about gain here - especially since for restore client goes opens connection to server, then server opens connection to storage node where index and then... simply too much connections across separate entities which will cause performance penalty and may cause of sorts of issues in case something is wrong with network.  Nevertheless, there is one distribution of index database which might be good.  Create index db on client itself.  Learn from Avamar.  Avamar keeps database local to client for example.  What is index database?  It is just collection of information of files and directories saved and kept inside as per retention applied to backup cycle.  OK, so client saves data, in temporary index you store data about saved files and directories, once all is done this temp index gets merged to real one on client.  When you do restore, no longer long browse times - it's all local.  Cool?  Not!  Reason why you backup the client is because of disaster so if that client goes down you loose index.  OK, but you backup index db to storage node.  Indeed you do, but how do you backup index db then?  You can't do it while backing up client as that's when index db is open... meaning it would probably not work well.  OK, you could do it always as last saveset (but think of all those users who have separate clients for separate save sets on same box and overlap mess that would create). Second, how do you get browsable restore when client gets compromised.  So, while I can understand you may have issue with index db space, current model seems to be better than any alternative I can think of.  If you really need to use local disks, use more than one at least - that would be my recommendation.