Can't get BMR working - Error trying to validate NetWorker server credentials

The full error is:

Error trying to validate NetWorker server credentials
No response from server XXXXXXX

Sorry for the late reply, we were finally able to test this successfully.

Thank you so much for your work, I would have never guessed it in a million years, what was EMC thinking there, these issues render the BMR ISO unusable out of the box.

So my guess is you deleted the conflicting ns peer information of the client on the nw server?

That is a peculiar design flaw that even though NW security prevents the BMR to act as the client with the same name, that the BMR wizrad does not mention this to take into account.

Just like bingo's document also stated, some teams have customized their nw bmr iso, adding things like multiple NW backup infra components into the hosts file for proper resolving.

Be aware that some older iso's had a java bug issue that it could not handle a date beyond Jan. 1st 2022. You would have to back date the BMR image date before that date (or set it in the VM's BIOS) to be able to even start the BMR wizard. That is fixed in more recent bmr iso's.

We also ran into another issue with the BMR approach, when dealing with remote locations, whenever selecting a specific moment to restore from, that then the whole client index needed to be moved from the backup server to the remote client (even though backup itself would come from local backup devices on the remote location). The worse the latency was, the longer it took before the actual restore would even start. So the more files a windows client had, the longer it could take. We've seen even that this part of the recovery process took more than 24h, especially when millions of files were involved. So this is the part before the actual restore even starts. There is no feedback whatsoever up until that point. No progress, no nothing. Seems simply to be hung. So while there is no issue at all to make a backup of a remote client to backup devices on the remote location and even having a remote NW storage node, regular BMR is severely impacted by this when having high latency (above 30-50ms it is noticeable, if it is above 100ms or higher, then it becomes abysmal). Needing to only be able to choose from full backups by specifying on CLI what ssid's needs to be restored, is cumbersome as you would not be able to have the most recent backup restored).

Thank you Barry for pointing out the other details.

With respect to your 'latency' problem I might have a clue, especially when I read statements like " So the more files a windows client had, the longer it could take.". Of course I do not know your configuration but if the save sets are stored on a remote disk device and "scanner" is used, such might happen.

I remember the very first Legato Tech Bulletin (more than 30 years ago) where a recovery was possible by piping the "scanner" output through the "recover" command. I am pretty sure that I could find it somewhere by looking in by "stone-aged archives"which I actually do not carry with me.

So - if "scanner" will really be used internally, the other big problem may arise which I throroughly described in my document https://avus-cr.de/gener785_eng.pdf - at least your symptoms point towards it. That would also explain why there is no feedback at all.

Just a guess ...

But although the problem has been verified, confirmed and been escalated Dell/EMC has not addressed it for mor than one year now.

in our case we have dozens of customer locations, each having a local DD and a local NW SN. SO clients backup to the local DD.

However the NW client indices are located on the NW backup server only, the very extreme long delay occurs when sifting through the NW client index. The more files, the longer it takes.

It was stated more or less as works like designed. However if NW would take the approach of other backup products also to be able to have the client index locally available on the NW SN that is on the same location as the clients, then reading through it might be mitigated.

Why this only shows up during a restore, is what is peculiar. The same can also be noticed whenever simply starting a recover locally, which also would have to sift through the same client index. Also when doing this on the NW SN on customer location, this takes way longer than on a NW SN that is very near to the NW server.

Some longer querying would be expected, however based on just the amount of the client index needing to be transmitted, also that is not where the issue would lie (maybe a couple of GB for each client index). So possibly many lookups or whatever are being performed (a restore doesn't even log at all what it does or what it is even waiting for).

We also simply tested this by running a nsrinfo for a client listing all files in the backup, which we ran on the NW storage node in the various locations. The worse the latency was, the longer the nsrinfo took. For a NW SN in the same locations as the NW server it took 45 seconds (on the backup server itself it took 30 secs), whereas for a location that had more than 120ms latency it took a whopping 55 minutes! And that was for just one backup for a client with 1.7 million files.

test to see if I can add a comment to a solved post?

was having issues doing so yesterday. so still seems allowed.

Here goes.

As we had an issue with BMR, I had a thorough look again at the issue stated in KB https://www.dell.com/support/kbdoc/en-ca/000196593/kb-196593-bmr-bare-metal-restore-wizard-fails-with-error-error-trying-to-validate-networker-server-credentials-no-response-from-server and the BMR error "Error trying to validate NetWorker server credentials No response from server XXXXXXX".

I tested this in my testlab with NW19.5.0.7 NW NVE server and a BMR recovery for a windows 2022 server VM. I only could get it to work if I set the actual date one day earlier. Been fiddling around with the date/time/timezone (the latter being reset to a default whenever closing the bmr wizard). Again and again the same error, with both nw19.5 and 19.6 BMR ISO.

However when I set the date a full day earlier, it continued.

Due to the lack of commands in the booted BMR client, I could not see what the timezone was it was being set to. Used "date /T" and "time /T" showing date and time that seemed to be OK. Also tried setting the time some minutes earlier. But still didn't work. Only once I set the date one full earlier, than actual date (so set to Oct 3rd while it was Oct. 4th), it worked.

Also I did not have to disable the local FW (wpeutil disablefirewall). Yes the NW server cannot ping the BMR client by default, but it does not seem to be necessary to do so to have a working BMR recovery (at least not with nw19.5 server and nw19.5+19.6BMR. Will see to test this with earlier BMR iso's as well.

So the only thing needed in my test lab was to set the date one day earlier. Nothing else.

To me this looks as if the timezone being set is doing something weird, due to which the actual date is in the future (as also described by the KB article but that seems to suggest simply to set the correct timezone, however that is NOT working when setting it to UTC+1 Amsterdam, didn't test other timezones), due to which the nsr peer certificate handshake is not approved by the NW server.

Still have to check in production if that indeed "solves" (or better "workaround") the issue.

As said will also try to test with earlier BMR iso's. Cannot test on earlier NW servers as we don't have any running anything lower than nw19.5.0.6.

So things seem to be a bit different compared to what bingo.1 experienced and stated in his PDF of how to address the issue by needing to disable the FW (which still might be valid with a nw19.3 server, however in my lab it was not needed at all with nw19.5 nw server).

Looking forward to results of others however...