Skip to main content
Start a Conversation

Unsolved

AS

Arne Steeg

1 Message

3953

April 21st, 2021 12:00

retrying mmd reservation request!!!

Hello Community,

i have the problem to clone my save sets. Sometimes it works. But usally i have to retry it. The log said always:

21.04.2021 17:59:55 Retry reservation after it was rejected. Retry counter [4]
21.04.2021 17:59:55 Sending cancel broker request 532
21.04.2021 17:59:55 Request 532 was cancelled.
21.04.2021 18:00:05 retrying mmd reservation request!!!
21.04.2021 18:00:15 Retry reservation after it was rejected. Retry counter [3]
21.04.2021 18:00:15 Sending cancel broker request 533
21.04.2021 18:00:15 Request 533 was cancelled.
21.04.2021 18:00:25 retrying mmd reservation request!!!
21.04.2021 18:00:35 Retry reservation after it was rejected. Retry counter [2]
21.04.2021 18:00:35 Sending cancel broker request 534
21.04.2021 18:00:35 Request 534 was cancelled.
21.04.2021 18:00:45 retrying mmd reservation request!!!
21.04.2021 18:00:55 Retry reservation after it was rejected. Retry counter [1]
21.04.2021 18:00:55 Sending cancel broker request 535
21.04.2021 18:00:55 Request 535 was cancelled.
21.04.2021 18:01:05 retrying mmd reservation request!!!
21.04.2021 18:01:15 Retry reservation after it was rejected. Retry counter [0]
21.04.2021 18:01:15 Sending cancel broker request 536
21.04.2021 18:01:15 Request 536 was cancelled.
21.04.2021 18:01:25 retrying mmd reservation request!!!
21.04.2021 18:01:35 Sending cancel broker request 537
21.04.2021 18:01:35 Request 537 was cancelled.
21.04.2021 18:01:45 retrying mmd reservation request!!!
21.04.2021 18:01:55 Sending cancel broker request 538
21.04.2021 18:01:55 Request 538 was cancelled.
21.04.2021 18:02:05 retrying mmd reservation request!!!
21.04.2021 18:02:15 Sending cancel broker request 539
21.04.2021 18:02:15 Request 539 was cancelled.
21.04.2021 18:02:25 retrying mmd reservation request!!!
21.04.2021 18:02:35 Sending cancel broker request 540
21.04.2021 18:02:35 Request 540 was cancelled.
21.04.2021 18:02:45 retrying mmd reservation request!!!
21.04.2021 18:02:55 Sending cancel broker request 541
21.04.2021 18:02:55 Request 541 was cancelled.
21.04.2021 18:03:05 retrying mmd reservation request!!!
21.04.2021 18:03:15 Sending cancel broker request 542
21.04.2021 18:03:15 Request 542 was cancelled.
21.04.2021 18:03:25 retrying mmd reservation request!!!

And i dont undestand what it means.

here the ful log:

07.04.2021 03:24:20 Step (1 of 16): NSRCLONE_INIT_PHASE_ONE: nsrclone has been started on host 'networker.cloudi.de', with command 'C:\Program Files\EMC NetWorker\nsr\bin\nsrclone.exe -a *policy name=CLone -a *policy workflow name=Clone Tresor VM -a *policy action name=clone -s networker -b Kopie-Tresor -J networker -y 1 Years -w 1 Months -o -F -S '.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ============== nsrcloneconfig parameters ===============================
07.04.2021 03:24:20 04/07/21 03:24:20.378616 max_threads_per_client(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 max_savesets_per_thread(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 max_client_threads(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 mmd_reservation_timeout(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 disable_vsr(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 concurrent_clone_enabled(1) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_enabled(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_slice_size_factor(31) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_preferred_slice_count(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_min_concurrent_slice_count(1) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_max_concurrent_slice_count(60) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ams_force_multithreaded_slicecopies(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 co_use_ip_if(0) is default value.
07.04.2021 03:24:20 04/07/21 03:24:20.378616 ========================================================================
07.04.2021 03:24:20 Step (2 of 16): NSRCLONE_INIT_PHASE_TWO: Obtaining nsrclone options.
07.04.2021 03:24:20 Step (3 of 16): NSRCLONE_INIT_PHASE_THREE: Performing Recover Pipe to Save disabled cloning.
07.04.2021 03:24:20 Action clone 'clone' has initialized as 'clone job' with job id 2151054
07.04.2021 03:24:20 Step (4 of 16): NSRCLONE_INIT_PHASE_FOUR: Validating the command line arguments.
07.04.2021 03:24:21 Obtaining all available input work items.
07.04.2021 03:24:21 Step (5 of 16): NSRCLONE_PROCESSING_PHASE_ONE: Submitting all save sets for Recover Pipe to Save processing.
07.04.2021 03:24:21 RPS is not enabled but VM savesets are detected, will use RPS framework for cloning of these savesets
07.04.2021 03:24:21 Step (6 of 16): NSRCLONE_PROCESSING_PHASE_TWO: Obtaining the save set ID and clone ID for the respective save sets.
07.04.2021 03:24:21 107783615, 124560830,
07.04.2021 03:24:21 141337888, 158115083,
07.04.2021 03:24:21 174892235, 191669388,
07.04.2021 03:24:21 208446384, 225223590,
07.04.2021 03:24:21 23897986, 242000785,
07.04.2021 03:24:21 258777980, 275554976,
07.04.2021 03:24:21 292332023, 309109164,
07.04.2021 03:24:21 325886380, 40675109,
07.04.2021 03:24:21 4201425267, 4218202461,
07.04.2021 03:24:21 4234979627, 4251756761,
07.04.2021 03:24:21 4268533944, 4285311078,
07.04.2021 03:24:21 57452175, 7120791,
07.04.2021 03:24:21 74229267, 91006461;
07.04.2021 03:24:21 Step (7 of 16): NSRCLONE_PROCESSING_PHASE_THREE: Found the recover storage node 'networker.cloudi.de' for source volume '5564328'.
07.04.2021 03:24:21 107783615/1617733054, 124560830/1617733054,
07.04.2021 03:24:21 141337888/1617732896, 158115083/1617732875,
07.04.2021 03:24:21 174892235/1617732811, 191669388/1617732748,
07.04.2021 03:24:21 208446384/1617732528, 225223590/1617732518,
07.04.2021 03:24:21 23897986/1617733506, 242000785/1617732497,
07.04.2021 03:24:21 258777980/1617732476, 275554976/1617732256,
07.04.2021 03:24:21 292332023/1617732087, 309109164/1617732012,
07.04.2021 03:24:21 325886380/1617732012, 40675109/1617733413,
07.04.2021 03:24:21 4201425267/1617734003, 4218202461/1617733981,
07.04.2021 03:24:21 4234979627/1617733931, 4251756761/1617733849,
07.04.2021 03:24:21 4268533944/1617733816, 4285311078/1617733734,
07.04.2021 03:24:21 57452175/1617733263, 7120791/1617733527,
07.04.2021 03:24:21 74229267/1617733139, 91006461/1617733117;
07.04.2021 03:24:27 Retry reservation after it was rejected. Retry counter [4]
07.04.2021 03:24:27 Sending cancel broker request 67
07.04.2021 03:24:27 Request 67 was cancelled.
07.04.2021 03:24:37 retrying mmd reservation request!!!
07.04.2021 03:34:48 Sending cancel broker request 68
07.04.2021 03:34:48 Request 68 was cancelled.
07.04.2021 03:34:48 retrying mmd reservation request!!!
07.04.2021 03:44:58 Sending cancel broker request 69
07.04.2021 03:44:58 Request 69 was cancelled.
07.04.2021 03:44:58 retrying mmd reservation request!!!
07.04.2021 03:55:19 Sending cancel broker request 70
....

....
07.04.2021 09:10:51 retrying mmd reservation request!!!
07.04.2021 09:11:01 Sending cancel broker request 310
07.04.2021 09:11:01 Request 310 was cancelled.
07.04.2021 09:11:11 retrying mmd reservation request!!!
07.04.2021 09:11:21 Sending cancel broker request 311
07.04.2021 09:11:21 Request 311 was cancelled.
07.04.2021 09:11:31 retrying mmd reservation request!!!
07.04.2021 09:11:41 Sending cancel broker request 312
07.04.2021 09:11:41 Request 312 was cancelled.
07.04.2021 09:11:51 retrying mmd reservation request!!!
07.04.2021 09:12:01 Sending cancel broker request 313
07.04.2021 09:12:01 Request 313 was cancelled.
07.04.2021 09:12:11 retrying mmd reservation request!!!
07.04.2021 09:12:21 Sending cancel broker request 314
07.04.2021 09:12:21 Request 314 was cancelled.
07.04.2021 09:12:31 retrying mmd reservation request!!!
07.04.2021 09:12:41 Sending cancel broker request 315
07.04.2021 09:12:41 Request 315 was cancelled.
07.04.2021 09:12:51 retrying mmd reservation request!!!
07.04.2021 09:13:02 Sending cancel broker request 316
07.04.2021 09:13:02 Request 316 was cancelled.
07.04.2021 09:13:12 retrying mmd reservation request!!!
07.04.2021 09:13:22 No more retries for reservation
07.04.2021 09:13:22 Quitting MMD reservation due to request status = REQ_REJECTED!!!
07.04.2021 09:13:22 Sending cancel broker request 317
07.04.2021 09:13:22 Request 317 was cancelled.
07.04.2021 09:13:22 NSRCLONE being aborted since broker request for mmd reservation failed with nonretryable error.
07.04.2021 09:13:22 Request 286 was cancelled.
07.04.2021 09:13:22 Failed to get mmd reservation with err: MMD reservation failed. .
07.04.2021 09:13:22 MMD reservation failed.
07.04.2021 09:13:23 No valid input savesets - check CLONE SKIPPED SAVESETS!
07.04.2021 09:13:23 RPS CLONE RESULTS \n [ORIGINAL REQUESTED SAVESETS]
07.04.2021 09:13:23 107783615, 124560830,
07.04.2021 09:13:23 141337888, 158115083,
07.04.2021 09:13:23 174892235, 191669388,
07.04.2021 09:13:23 208446384, 225223590,
07.04.2021 09:13:23 23897986, 242000785,
07.04.2021 09:13:23 258777980, 275554976,
07.04.2021 09:13:23 292332023, 309109164,
07.04.2021 09:13:23 325886380, 40675109,
07.04.2021 09:13:23 4201425267, 4218202461,
07.04.2021 09:13:23 4234979627, 4251756761,
07.04.2021 09:13:23 4268533944, 4285311078,
07.04.2021 09:13:23 57452175, 7120791,
07.04.2021 09:13:23 74229267, 91006461;
07.04.2021 09:13:23 [CLONE SKIPPED SAVESETS]
07.04.2021 09:13:23 107783615/1617733054, 124560830/1617733054,
07.04.2021 09:13:23 141337888/1617732896, 158115083/1617732875,
07.04.2021 09:13:23 174892235/1617732811, 191669388/1617732748,
07.04.2021 09:13:23 208446384/1617732528, 225223590/1617732518,
07.04.2021 09:13:23 23897986/1617733506, 242000785/1617732497,
07.04.2021 09:13:23 258777980/1617732476, 275554976/1617732256,
07.04.2021 09:13:23 292332023/1617732087, 309109164/1617732012,
07.04.2021 09:13:23 325886380/1617732012, 40675109/1617733413,
07.04.2021 09:13:23 4201425267/1617734003, 4218202461/1617733981,
07.04.2021 09:13:23 4234979627/1617733931, 4251756761/1617733849,
07.04.2021 09:13:23 4268533944/1617733816, 4285311078/1617733734,
07.04.2021 09:13:23 57452175/1617733263, 7120791/1617733527,
07.04.2021 09:13:23 74229267/1617733139, 91006461/1617733117;
07.04.2021 09:13:28 The nsrclone verification detected some failures.
07.04.2021 09:13:28 04/07/21 09:13:28.296513 Save set list count: 26, successful count: 0, failed count: 26
07.04.2021 09:13:28 04/07/21 09:13:28.296513 Total save set list: : 91006461, 74229267, 7120791, 57452175, 4285311078, 4268533944, 4251756761, 4234979627, 4218202461, 4201425267, 40675109, 325886380, 309109164, 292332023, 275554976, 258777980, 242000785, 23897986, 225223590, 208446384, 191669388, 174892235, 158115083, 141337888, 124560830, 107783615
07.04.2021 09:13:28 04/07/21 09:13:28.296513 Failed save set list: : 91006461, 74229267, 7120791, 57452175, 4285311078, 4268533944, 4251756761, 4234979627, 4218202461, 4201425267, 40675109, 325886380, 309109164, 292332023, 275554976, 258777980, 242000785, 23897986, 225223590, 208446384, 191669388, 174892235, 158115083, 141337888, 124560830, 107783615
07.04.2021 09:13:28 04/07/21 09:13:28.296513 Cloned save set list::
07.04.2021 09:13:28 Action clone 'clone' with job id 2151054 is exiting with status 'failed', exit code 1
07.04.2021 09:13:28 NSRCLONE failed for one or more savesets.

1 person also has this problem

bingo.1

2.4K Posts

April 21st, 2021 14:00

Hello,

before I am going to discuss the problem let me just start with some fundamental statements

  • Why do you use the system partition to install NW? - I admit it must be there but it is the worst case if you have problems with the OS itself. Please use separate volumes for your apps &data.
    • If your system is not already to big, you better change that NOW.
  • Which exact OS version are you using it? - only from the pathname we see that it must be Windows.
  • Which NW version do you use at all?
  • Give us a brief description about your HW setup.
    • How many remote storage nodes do you use (at all)?
    • Where do you backup to? - type and number of devices.
    • Where do you clone to? - type and number of devices.
  • A personal hint: Do not use clone actions at all, especially at the end of a backup workflow. You will find the reasons below.

 

Now with respect to your technical problem.

nsrmmd processes (aka Media Management Daemon) deal with the backup device. There is at least one nsrmmd process active on the storage node for each active device, even if it does not do anything. More may apply depending on the number of sessions and the NW version and other parameters. So if NW cannot get not start a new nsrmmd for whatever reason, it will retry in a few seconds. And if the system is still busy - for example with other backups - it just cannot clone to the same device/volume, especially if the device does not allow read/write processes from the same volume at the same time (like tapes).

The next issue is that for troubleshooting you better rely on the command line. This has a lot of benefits:

  • You exactly see and control what NW is doing.
  • You can easily modify and repeat a command, if necessary, without disturbing your configuration.
  • You avoid 'translation errors' which occur when you use the GUI. As you can see, these exist.
  • You can just use these commands in (Power)shell scripts and start them independently at times when no other process uses the backup device and/or the media - especially when no other workflows try to load the devices/volumes for other purposes.

As you can see, NW itself relies on the command line. He is using the nsrclone command with these parameters. Please use the Command Line Reference or the nsrclone -? command for assistance:

clone -s networker -b Kopie-Tresor -J networker -y 1 Years -w 1 Months -o -F -S

Here are the problems:

  • You do not see the command itself - it is nsrclone
  • According the the docs the sorting parameter -o needs the option how to sort. This is simply missing here. I do not know whether this might cause a problem.

So I would like you to do the following:

  • Use the mminfo command to get a list of all the save sets ids and store them into a file. The core command should look like
    • mminfo -q "your query" -r "ssid" > file  . I am especially not more precise because you must use mminfo all the time. You better understand the command asap (not all options, that's why you have the docs).
  • Now modify your nsrclone command as follows
    • nsrclone -s networker -b Kopie-Tresor -J read_storage_node -d write_storage_node -y "1 Years" -w "1 Months" -F -S -f "absolute_path_name_to_your_ssid_file" > "output_file.txt" 2>&1
  • Run the commands when the server is idle.
  • Verify the output file.

This is enough work for now . Good luck.

 

JChan2020

18 Posts

April 26th, 2021 03:00

Hi. I encountered a similar issue when cloning backup from DD to tape.

Please see if it is your case.

From the Defects / Escalations List of NetWorker 19.2.1.4,

"326991 NetWorker Escalation 39194: DD to tape clone fails error: Failed to get mmd reservation with err: MMD reservation failed"

barry_beckers

393 Posts

May 3rd, 2021 13:00

therefor it is always handy to state what the nw version used is?

whenever someone runs into a specific error which doesn't ring any bells right away, one of the first things I do is to try look it up in the appropriate NW_xx.x.x-xx.x.x_Recommendations_Cumulatives_Training_Downloads pdf to see if anything similar is listed there to be fixed already.

However sometimes it makes me wonder? If I can lookup an error in those documents on https://www.dell.com/support/home/en-us/product-support/product/networker/drivers, anyone (with a Dell support login) can do that as well? However it isn't that common I can tell you...

bingo.1

2.4K Posts

May 5th, 2021 14:00

o.k. guys,

having retired a while ago I tried to verify the issue. Obviously, I have more time than those of you who still have to deal with the daily business. The results of my investigations are pretty weird but I will discuss those a bit later.

To give Dell/EMC the chance to verify the problem, here are the fundamental facts about my environment:

  Windows 2019 server (NW server)

  NW 19.4.0.0 - Build 25

  Backup is using the pool Default with only one DDBoost medium

  Clone is handled by a clone workflow sending the save set to a LTO4 drive.

  Besides that, NW is using the default parameters.

Sorry guys, but such is the information which you should provide whenever you start a thread ...

 

Due to the error I expected a problem which appears when there is a backup running while a clone workflow is active. In fact, I expected this to happen already with a single save set. So I

  - created a full backup of a 60 GB save set

  - started a second backup of the same backup

  - as soon as the data transfer started, I triggered the clone workflow to execute

The result surprised me:

  - There was no functional problem

  - However, in the end the NW media index showed 4 entries for the cloned save set instance:

C:\>mminfo -q "savetime>05/03/2021" -r "name,client,level,savetime(25),sumsize(15),ssid,cloneid,ssinsert(25),sscomp(25),sumflags,volume" -ot -xc,

 

name,client,level,date-time,sum-size,ssid,clone-id,ss-inserted,ss-completed,flags,volume
U:\,dcon,full,05/05/2021 12:57:42,61833277376,2995943976,1620212264,05/05/2021 12:57:44,05/05/2021 13:08:27,cb,Default.001
U:\,dcon,full,05/05/2021 12:57:42,61833277376,2995943976,1620218166,05/05/2021 12:57:44,05/05/2021 13:08:27,cb,Tape_Clone.002
U:\,dcon,full,05/05/2021 12:57:42,61833277376,2995943976,1620218166,05/05/2021 12:57:44,05/05/2021 13:08:27,cb,Tape_Clone.002
U:\,dcon,full,05/05/2021 12:57:42,61833277376,2995943976,1620218166,05/05/2021 12:57:44,05/05/2021 13:08:27,cb,Tape_Clone.002
U:\,dcon,full,05/05/2021 12:57:42,61833277376,2995943976,1620218166,05/05/2021 12:57:44,05/05/2021 13:08:27,cb,Tape_Clone.002
U:\,dcon,full,05/05/2021 14:21:13,61831867600,2945617340,1620217276,05/05/2021 14:21:16,05/05/2021 14:47:08,cb,Default.001





C:\>

Such is confusing because in large systems it could easily blow up the media index with redundant information. However it does not really harm as all entries show the same information.

###############################################################################

Because I vaguely remembered an issue with this parameter some years ago, I now set the server parameter Disable RPS Clone to unchecked ...

Versuch_4_-_2.jpg

... and I repeated the test procedure. Surprisingly, this worked without problems.

 

Next I put a bit more load on the system and backed up 3 save sets which I repeated and cloned using the clone workflow as before. Suddenly there it was - the mmd reservation conflict!

 

Here is my conclusion. All seems to be a combination of

  - using a clone workflow

  - along with the option Disable RPS Clones (yes - by default it is checked but if you upgrade from older NW versions, this might be different)

  - the load (number of active save sets)

It could be that devices Target Sessions would also be involved but I did not test that so far. I just want to keep the scenario as simple as possible.

 

How to overcome the issue:

  - Do not use clone workflows at all (some of you know that I am a fan of scripted cloning)

  - Double-check the option Disable RPS clones.

 

I hope that this will help you guys who suffer from this issue.

 

barry_beckers

393 Posts

May 9th, 2021 16:00

Thank you Bingo.1, for spending your precious time being retired already! Dunno if I'd be able to keep on contributing to the community once I'd made it that far?

RPS being enabled or not has caused many issues in the past. Mainly we left it disabled. Rps was used regardess when cloning vproxy vm backups. However not being able to control it for all possible scenarios in a landscape could become very cumbersome, required for one thing and advised against for another without being able to differentiate further (for example giving it a specific option for cloning towards tapes, while being able to use the opposite for datadomain (while vproxy always uses rps regardless). So much for control.

Based on your test, when cloning towards tapes, rps should be enabled (so by leaving "disable rps cloning" unchecked).

At least, in your case the nsrmmd reservation  issue occurred under load while having the "disable rps clone" not checked? So that means rps is enabled.

KB https://www.dell.com/support/kbdoc/en-us/000065879/networker-18-instant-cloning?lwp=rt states that with rps enabled multiplexing would be used towards tape. If rps is not used it causes clone failures with error "6220:nsrclone: nsrclone command has retried 2 times."  

So using rps while cloning to tape might be beneficial in this case. However pay attention to the "advice" from the kb article for the used target and max sessions, keeping them very low to prevent overload. I don't know about you, but that - to me - doesn't sound like a very well scalable enterprise product, using that kind of advice in a KB article? Come on Dell...

"Cause

With Disable RPS Clone marked, the instant clones to tape will not multiplex.
Resolution

Unmarking "Disable RPS Clone" in the NetWorker server's "Server properties" will allow clones to multiplex to tape drives. RPS clones will allow clones to be multiplexed.

CAUTION SHOULD BE USED, NetWorker can over allocate resources when servicing RPS clones to the point that the NetWorker server will become unresponsive. Tune the tape drives' target sessions and max target sessions to not overrun the NetWorker server. Use target sessions 1 and max target sessions 2 for starters."

barry_beckers

393 Posts

May 9th, 2021 16:00

As another poster stated, not only for nw19.2 but also within nw19.4.0.1 that are multiple cloning related issues solved. besides those also the nsrdr related ones would be very good to address. so better to update 19.4.0.0 to 19.4.0.1 and depending on the setup 19.4.0.2 might be required even (Escalation 40330: After upgrade to NetWorker 19.4.0.1 we no longer can complete VMWare Hypervisor inventory)

 

Some (but not all) fixes introduced in nw19.4.0.1:

326991 NetWorker Escalation 39194: DD to tape clone fails error: Failed to get mmd reservation with err: MMD reservation failed

327534 NetWorker Escalation 39577: Max target session configuration is not been taking in consideration during clone operations

327630 NetWorker Escalation 39661: nsrdr command never finish and lead in OS hang, showing thousands of lines about "nsrworkflow"

327324 NetWorker Escalation 39310: nsrdr core dump when services respond for first and can't respond later after the bootstrap recovery

328169 NetWorker Escalation 39693: Clone from DD to tapes - Not assigning correct number of sessions if you set Max sessions 1

327285 NetWorker Escalation 39417: backup is marked as "complete/successful", despite unrecoverable read file error

bingo.1

2.4K Posts

May 10th, 2021 07:00

OK - i will upgrade to NW 19.4.0.2 (which has just been released over the weekend) and retry the tests ... later this week.

 

bingo.1

2.4K Posts

May 11th, 2021 11:00

Hello,

as promised I have upgraded my tiny test environment with NW 19.4.0.2 and repeated my tests with no other change in the test parameters.

This is to confirm that with this NW version, all seems to run fine - the backup jobs perform while the clone workflow is also running fine - at least according to the screenshot:

NW_19_4_0_2.jpg

 

So it seems that it is time to upgrade...

Good luck and best regards.

 

guillaume.leonard

1 Rookie

 • 

27 Posts

June 30th, 2021 07:00

Hi,

I don't know if bumping this thread is the correct way to do it. I am having the exact same issue, but I am running Networker 19.4.0.2 already. What is weird, is that the issue is happening on 2 Networker instances, completely separated/isolated from each other. Both are at 19.4.0.2.

Back in May, few days after 19.4.0.2 got released, I upgraded both of our Networker to 19.4.0.2 from 19.2.0.1.

The way that our backups work is that every day, we backup to a Data Domain. We have a few physical servers and the rest is all backed up through a vProxy appliance. The dailys are running fine.

Every last Friday of each month, we take a full backup of all our infrastructure and clone that backup to LTO7 tapes. I upgraded Networker to 19.4.0.2 mid-May and the cloning job at the end of May worked successfully. Last Friday however, June 25th, the cloning process failed on both of our Networker instances. There have been some changes on one of the Networker instance (had to use nsrdr few days ago), but the other Networker instance had no changes since the upgrade to 19.4.0.2 back in May. So, 19.4.0.2 cloning from DD to tapes worked at the end of May, but not at the end of June.

To add more, the cloning of physical servers always work, but not the vProxy. When the cloning reaches the clone save sets from the vProxy, then I get the infamous retrying mmd reservation request!!!

At this time, the cloning process is still part of the workflow running after the backups, but I'm working to get the cloning scripted using nsrclone. I tried to clone using nsrclone instead of through the workflow in order to maybe bypass the issue, but I still get retrying mmd reservation request!!! when getting to clone VMs using nsrclone in a command line.

I have read on this forum about disabling/enabling RPS. It didn't do anything for me.

I of course tried multiple reboots of the server and Networker services. No luck either.

Any pointers ?

I can provide logs if necessary but I'll have to trim a bit, since they are quite big.

Thank you.

bingo.1

2.4K Posts

July 1st, 2021 12:00

Maybe the new NW version 19.5.0.0 which has been released yesterday (06/30/2021) will finally solve the issue.

In the Release Notes, there is this new feature listed: Improved concurrency control for cloning using action parallelism

 

guillaume.leonard

1 Rookie

 • 

27 Posts

July 5th, 2021 12:00

Thanks bingo.1. So far so good...after the upgrade, I didn't change any settings and nsrclone is working now.

Let's hope it will stay that way.

bingo.1

2.4K Posts

July 6th, 2021 01:00

So did you already install 19.5.0.0 yet? - And you say it's doing the trick (at least in your environment) ...

 

bingo.1

2.4K Posts

July 6th, 2021 05:00

Thank you so much for sharing this information.

 

guillaume.leonard

1 Rookie

 • 

27 Posts

July 6th, 2021 05:00

Yep ! Sorry if that wasn't clear. I upgraded to 19.5.0.0 yesterday and it's now working on my 2 Networker instances, without any configs changes.

bingo.1

2.4K Posts

July 9th, 2021 09:00

Today I could update my test environment with NW 19.5.0.0 and just confirm that these nsrmmd reservation conflicts are gone now. Obviously the problem occurred because 2 processes (backup & clone) were accessing an unfinished save set at the same time.

If you now (NW 19.5.0.) start the clone workflow and there is not save set finished yet, the workflow will run and end successfully but the list of cloned save sets will be empty.

 

View More

View All

No Events found!

Top