Ask the Expert: NetWorker Module for Microsoft (NMM) - Configurations, Best Practices and Solutions

Hello all,

Welcome to this session of "Ask the expert".

As I don't see yet any question, I'd like to start by sharing the compatibility matrix, which covers the compatibility of all NW client, storage node and NW server versions with the different releases of NMM.

Please see below the screenshot of the support matrix.

Thank you,

Carlos

Hi Carlos,

Couple of question, all related to SQL backups/restore using VDI to kick the discussion in.

1. What is better (and suggested): to run backup on SQL instance level or DB level?  Obviously, it is customer call, but my question goes more against performance and load and any gotchas there might be.  For example, if I have SQL hotel (SQL box hosting couple of instances where each instance may host 20-50 databases), having SQL TLOG groups which runs each 1h might cause some overlaps which with instance level might not be due to sequential nature of execution and so on.  What is the best practice there? (obviously there might be also an issue with jobdb going nuts with increased number of streams/sessions to take care of)
2. Since moving to NW8 (server wise) with different cases on clients tested so far (NMSQL 5.2.2b23&29 + NW7 and 8, NMM 3.0 and NMM 3.1 using 8.1) I have seen that many times nsrexecd on client will freeze - stop responding to RPC queries.  When this happens, rpcinfo -t will give you time out and it may go on like this for hours and hours.  Only solution is to restart nsrexecd service. I could not test if changing backups to instance level opposed to individual DB has effect (see 1), but spreading some backups does help. I wonder if anything like has been observed and what would be suggestion here.
3. One thing that one can easily see is that browsing using NMM (and NMSQL previously) GUI does take ages.  While I never debug this, I assume client does build list of all backups and this takes long time (imagine box with 50 databases where apart from 1 DB backup you have TLOG every 1h so you easily end up with many many index entries - and backup is done on individual DB basis). With CLI, this goes much much faster - as you provide -t immediately (in GUI this is first built up for you to select).  Is there any fix to this?  How does this translated to NMM 8.2?  I never tested it, but there you can use SQL mgmt studio to do restores - does this work faster or does it still use NW server to build index/timestamp tree?
4. In respect to NMM 8.2 and integration with SQL mgmt studio; can you comment following:
   • NMM plugin backups are VDI or VSS based?
   • if backup is done with NW+NMM, can this be restored using SQL mgmt studio GUI
   • if backup is done with SQL mgmt studio and NMM plugin, can this be restored with NMM GUI
5. If using VDI approach for SQL backup, is it recommended to disable SQL writer during FS backups with NW (to avoid any sort of conflict)?
6. I believe 8.2 does use its own RPC service if I read this correctly - not sure if this is general feature or UNIX only.  If applicable to Windows, how does this impact NMM?  (obviously NMM has to same release as client for start, but what about workflow itself)
7. Some recent releases of NW clients (builds) appear to have NMM releases following those builds too.  Unlike NW, NMM does not have public "download and apply" location for these patches - is this something that will change soon?  Will users have ability to download patches just as they can do for NW and see fix list?

This is top of my head.  Thank you

Hi Carlos, first of all, thanks for this opportunity! I bumped into this article today.

I would like to know the best settings in NetWorker when need to perform backups of Exchange 2007. Here are some details.

-It's a 2 node cluster running Windows 2008R2 (virtual)

-All active 30 SG's on 1 node, passive ones on the other (CRR)

-Capacity of all SG's combined: 5TB

-Config checker runs clean

-NetWorker 8.1.1.6

-NMM 3.0SP1

-Backup to DD4500 via Boost

The NMM guide talks about splitting up the backup in 10 SG's at a time. This means that I would have to create 3 groups and schedule them in a correct order. All because of the consistency check that runs in parallel which is eating up resources. I'm looking for a good way to run this in 1 group and complete the backup within a standard backup window. I know this can be achieved with IO throttling but I'm looking for some more detail. What would you advise? What would your steps be in configuring this?

Like to hear your thoughts on this.

Best regards,

-Nick

Hello Hrvoje Crvelin 

Thank you for starting the discussion, and of course I wouldn't expect nothing but very good questions from you

See below my answers inline, and please don't hesitate to ask for further clarifications:

1. What is better (and suggested): to run backup on SQL instance level or DB level?  Obviously, it is customer call, but my question goes more against performance and load and any gotchas there might be.  For example, if I have SQL hotel (SQL box hosting couple of instances where each instance may host 20-50 databases), having SQL TLOG groups which runs each 1h might cause some overlaps which with instance level might not be due to sequential nature of execution and so on.  What is the best practice there? (obviously there might be also an issue with jobdb going nuts with increased number of streams/sessions to take care of).

[CR]: This is kind of a tricky one; as you said, this is customer's call, but I personally would recommend to use the most granular level for backups, the dangerous side of this approach is when the DBA creates a new DB and the backup admin is not notified, in that case that DB is not being backed up unless the backup admin manually adds the new saveset.

In order to meet those demanding RTO's and RPO's, you can spread the DB's across different groups so that there is no overlapping of the full and incremental (Tlog) backups.
You get better performance with DB level in terms of when the group gets started, as with the instance level it will have to discover all the DBs contained by that SQL instance, but you are safe in regards to the DB's created in that instance.

BUT this all depends on resources available, such as devices, storage nodes etc. If you have multiple devices, or even the SQL server is a storage node itself, with plenty of devices, then you can play around with the striping, which is what eventually will give you better performance and throughput, although it will have some impact in restores as well, talking about PTL/VTL, but if we are talking about Data Domain DDBoost backups, then striping is definitely what you should concentrate in, as this will allow to send more files concurrently and, as this is backup to disk, the throughput will be much better, and restore will be impacted in terms of speed, but yet will be much faster than backup to tape.
If you run the backup at DB level, it will spawn multiple nsrsqlsv.exe processes (up to 8, which is the hardcoded limit), and this will also improve the performance and will introduce less overhead on the SQL server.

2. Since moving to NW8 (server wise) with different cases on clients tested so far (NMSQL 5.2.2b23&29 + NW7 and 8, NMM 3.0 and NMM 3.1 using 8.1) I have seen that many times nsrexecd on client will freeze - stop responding to RPC queries.  When this happens, rpcinfo -t will give you time out and it may go on like this for hours and hours.  Only solution is to restart nsrexecd service. I could not test if changing backups to instance level opposed to individual DB has effect (see 1), but spreading some backups does help. I wonder if anything like has been observed and what would be suggestion here.

[CR]: There is currently an open escalation for another customer and it's yet being worked by engineering.
On the other hand, do you have multiple IP's environment? If so this could be normal as it will be checking all possible names and IP's.

3. One thing that one can easily see is that browsing using NMM (and NMSQL previously) GUI does take ages.  While I never debug this, I assume client does build list of all backups and this takes long time (imagine box with 50 databases where apart from 1 DB backup you have TLOG every 1h so you easily end up with many many index entries - and backup is done on individual DB basis).

With CLI, this goes much much faster - as you provide -t immediately (in GUI this is first built up for you to select).  Is there any fix to this?  How does this translated to NMM 8.2?  I never tested it, but there you can use SQL mgmt studio to do restores - does this work faster or does it still use NW server to build index/timestamp tree?

[CR]: Some fixes and improvements were introduced in NMSQL for this matter, but it's true that in some cases NMSQL GUI takes time to open, and that is, as you well described because of NMSQL has to make a connection to the CFI (Client File Indes) for that SQL client, and with such huge amount of number of backups then it will take longer indeed, although should be much less than in the past.

In regards to NMM 8.2, the behaviour is exactly the same, as it will also check the indexes, available backups for that SQL client, etc., however, with NMSQL GUI you can improve that behaviour by adding -b option at the end of the shortcut (or running it through command line) in order to instruct the client to check only for the number of days you want to go back to check the indexes for that client, for example if we only want to check the last 4 days:

Starting NMSL GUI from command line:

C:\>"E:\Program Files\EMC NetWorker\nsr\bin\nwmssql.exe" -b 4

Or just change the shortcut and add exactly the same as above:

"E:\Program Files\EMC NetWorker\nsr\bin\nwmssql.exe" -b 4

You can modify that value

4. In respect to NMM 8.2 and integration with SQL mgmt studio; can you comment following:

o NMM plugin backups are VDI or VSS based?
   [CR]: These are VDI based

o if backup is done with NW+NMM, can this be restored using SQL mgmt studio GUI

   [CR]: Yes, you can restore any backup that has been indexed, as NMM plugin will check the index to verify available backups for restore.

o if backup is done with SQL mgmt studio and NMM plugin, can this be restored with NMM GUI

   [CR]: Yes, same as above. Manual backups will create indexes that will be used later on for restore from either command line, NMM SQL GUI or SQL management studio plugin.

5. If using VDI approach for SQL backup, is it recommended to disable SQL writer during FS backups with NW (to avoid any sort of conflict)?

[CR]: No need to disable SQL writer, as there are internal directives to avoid backing up any SQL Writer related file, but no harm in disabling it either.

6. I believe 8.2 does use its own RPC service if I read this correctly - not sure if this is general feature or UNIX only.  If applicable to Windows, how does this impact NMM?  (obviously NMM has to same release as client for start, but what about workflow itself)

[CR]: Basically with NW 8.2 RPC services are detached from OS RPC, so there is not real change at workflow level, but only internally, to improve the RPC dependencies. Also note that with NW 8.2 rpcinfo command will NOT show NetWorker processes, but instead will have to use nsrrpcinfo (more informaiton in the administration guide).

The official description is as follows:

In NetWorker 8.2, NetWorker daemons do not register with system port mappers.
NetWorker daemons register exclusively with the port mapper that the nsrexecd process manages. As a result, the OS rpcinfo command will not display NetWorker daemons.

7. Some recent releases of NW clients (builds) appear to have NMM releases following those builds too.  Unlike NW, NMM does not have public "download and apply" location for these patches - is this something that will change soon?  Will users have ability to download patches just as they can do for NW and see fix list?

[CR]: You are right, there is not such public FTP where to download the fixes from, as these are usually more complex than NW server/client, so usually these fixes are provided through support after a proper assesment, to make sure the fix is going to help or solve the issue presented.
Same thing with the latest cumulative versions for NMM, these are usually handed over by support, but I'll double-check and will get back to you shortly.

Thank you,

Carlos

Hi Carlos,

Yes, consistency check will have to be run.

-Nick

Hello Nick

Thank you for posting your questions here,

When it comes down to configurations, there are general guidelines or best practices, but every environment is different.
In this case, I would like to know if you are running consistency check during the backups or not? As this would change my recommendations.

In any case, more than 10SG at the same time is not a good practice as you can end up with Exchange server being overloaded and even hanging, which will have impact in production.

Thank you,

Carlos

Unable to configure NMM clients using NMC client Wizard

Environment:

NetWorker Module for Microsoft (NMM)

Exchange Server

Hyper-V Server

SharePoint Server

SQL Server

Description:

When using NMC Client Wizard to create NMM clients, the wizard is not showing any valid savesets for the one or more MS application.

In some cases, wizard is not displaying even the File System savesets.

This could be because NMM silent Installation method was used to install NMM or because of a failure in registering the Remote Agent components of NMM.

Resolution:

First make sure all writers information is correctly populated.

To do it so run:

vssclupdate

Then please see below how to manually register the Remote Agent for NMM for all Microsoft applications and File System:

To register MS Exchange Server Remote Agent run on the Exchange server:

nsrnmmexchra -i

To register MS Hyper-V Server Remote Agent run on the Hyper-V server:

nsrnmmhypervra -I

To register MS SharePoint Server Remote Agent run on the SharePoint server:

nsrnmmspra -i

To register MS SQL Server Remote Agent run on the SQL server:

nsrnmsqlra -i

This is should fix the problem, however I would recommend to first unregister the RA and then register it again.

To unregister MS Exchange Server Remote Agent run on the Exchange server:

nsrnmmexchra -u

To unregister MS Hyper-V Server Remote Agent run on the Hyper-V server:

nsrnmmhypervra -u

To unregister MS SharePoint Server Remote Agent run on the SharePoint server:

nsrnmmspra -u

To unregister MS SQL Server Remote Agent run on the SQL server:

nsrnmsqlra -u

Then register it again with the same commands showed above (with -i option).

If the problem would be with the File System savesets, run the following commands:

nsrfsra -u (to unregister it)

Then
nsrfsra -i (to register it).

If this doesn't solve the problem it could be related to name resolution, especially in clusters.

Reference:

EMC Support Solution
Number: 000185178

Hi Carlos,

Thank you for your answers.  Comments:

1) All clear.

2) Yes, this is multihomed environment an backup goes over VLAN which is backend and it is isolated (so backend address can't communicate any kind of front-end address).  Client definition is based on back-end FQDN, storage node is defined as storage node name on back-end (FQDN) and backup server interface is based on server name on back-end FQDN address.

3) -b sounds interesting - will have to test it.

4), 5), 6), 7) - clear.

Hi Carlos, what would your recommendations be if I run the consistency check?

-Exchange 2007

-It's a 2 node cluster running Windows 2008R2 (virtual)

-All active 30 SG's on 1 node, passive ones on the other (CRR)

-Capacity of all SG's combined: 5TB

-Config checker runs clean

-NetWorker 8.1.1.6

-NMM 3.0SP1

-Backup to DD4500 via Boost

Thanks!

-Nick

Hi Hrvoje Crvelin,

About point 3 (long time to browse with NMSQL GUI), there is a fix released very recently that maybe you would like to ask support to provide it to you. Reference number is NW161065.

Thank you,

Carlos

Thanks Carlos.  I was actually given this one before I left for vacation so I need to test it too.  Thnx!

Hello Nick

In this case I don't see any other option but to create 3 groups to backup 10 SG on each group.

Being a CCR environment, it's up to the customer to backup the passive or the active DB's but, as consistency check is required, I would recommend to backup the passive DB's to avoid any performance issue on the active node.

I would also recommend to set some throttling to have less impact on the passive node, and for that reason I would suggest to set the following variables in the Application Information field (if creating the client manually) or set the following options (if using client wizard):

NSR_ESE_UTIL_SEQUENTIAL=True

NSR_ESE_UTIL_THROTTLE=True

NSR_ESE_THROTTLE_IOS=100

NSR_ESE_THROTTLE_DURATION=1000

With these settings we will achieve the following:

Consistency check will run in sequentially, to avoid any overhead on the node where the consistency check will run, but will increase the time for the consistency check completion, so will also increase the duration of the backup. This all depends on the HW resources on the Exchange server.

We will use throttling, to accelerate the consistency check, running 100 I/Os between pauses, and having pauses of 1 second.

If the Exchange server is very powerful you can play around with the values, and even setting consistency check to run in parallel instead of sequentially.

Thank you,

Carlos

Hi Carlos Rojas, thanks! This is helpful.

-Nick

Hi Carlos, we have some MSSQL 2008 DBs running Simple Recovery Mode.  Does Networker support differential backups for this recovery type?  I don't want to run a Full Backup every day.  If yes, what type of schedule can i use? Like Fri - Full  Sat - L2 Sun - L3 M -L4 Tue - L3 Wed - L4 Thur L5.