We have some trouble with a tape library managed by networker V19.4. and ealier versions.
After clone to tape jobs to a tape pool are finished, networker will not mount tapes from another tape pool to the used drive. If you mount a tape from the desired pool manually, the waiting job will start.
This happens with or without pool to target device selections.
First thing I've found, was that the tape storage node (bagsv32) was not listed under StorageNodeProperties -> Configuration -> Advanced Devices -> Clone storage nodes. After adding it there it was possible again to use the tape devices.
Second source of the trouble was a parallel installed networker client on the tape storage node (in addition with a different version). Since I have removed it, everything runs fine again.
Thanks to all the supporters in this thread for helping me!
This sounds weird. I personally expect a configuration error. In principle you are following the right idea - do not allow assign pools to devices.
I personally have never seen a good reason to use it. IMHO, this is the last choice because you can easily end up in a mess and cause confusion. Do not forget: "enablers are also disablers." - If you specify a certain criteria you block others.
I would use the "nsrjb -vvv -l ..." command to load and mount a tape in one of these devices. Now NW should tell you more where the trouble is. You could also look at the daemon.raw file.
11/18/20 08:25:07.669553 Error: Cannot allocate `1' devices for this operation: max active devices for jukebox rd=bagsv32.xxx.group:IBM@0.3.1 = 0
It is a TL4000 library attached to a w2016 server.
Mounting using the gui is working without problems.
But that seems not be the reason of all the trouble.
I tried a clone like this:
nsrclone -vvv -b Wochenband -S 2813537987
It gave me :
180206:nsrclone: Step (7 of 16): NSRCLONE_PROCESSING_PHASE_THREE: Sending the media reservation request for the current save set list.
160812:nsrclone: Retry reservation after it was rejected. Retry counter [4]
116284:nsrclone: Sending cancel broker request 96
180876:nsrclone: Request 96 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
After that I checked the 'Disable RPS Clone' box in the server properties and that solved the problem!
Instead of the error message the job continued with:
180179:nsrclone: Updating the total number of steps from 16 to 5 for the non VM save sets workflow.
Now the library is behaving like expected... But I don't know why the RPS option is causing this errors.
Well done. There are obvious still issues with this option although it is even enabled by default. But you obviously disable it when you so not clone between 2 DDs.
Unfortunately it turned out that the problem is still existing...
After starting another clone job with debug level 9 I have noticed this lines:
broker_find_usable_save_mmd: no ready or setup mmd available on snode bagsv32
is_device_busy: device rd=bagsv32:\\.\Tape1 assigned to mmd #269893653.
is_device_busy: device rd=bagsv32:\\.\Tape0 is not shared
find_init_mmd_for_save: write_reserved_mmds: 0
0 out of 8 mmds are available for save
broker_find_usable_save_mmd: NSRATTR_CLONE_SAVE_VOL_HINT not set in the request
Failed to allocate save mmd since it was not started. Reject request id 595
There are no available devices for client `bagsv32' on all usable storage nodes for the requested save operation.
Where is the problem? Should Tape0 be shared? Or is the number of possible mmd processes somewhere limited?
So all of the allowed nsrmmd processes have been started for this device and are in use. A new one to support the save/clone operation cannot be started.
This points to the fact that you either hit a limit (Max nsrmmd count - although it seems to be only valid for AFTD and DD devices) or that earlier nsrmmd processes have not been properly terminated.
After deleting and recreating the staorage node with the library attached it is no longer possible to clone to the library. Inventory of the tapes worked, but I cannot restore fronm tape or clone to tape any longer.
Unable to start clone session: no matching devices for save of client `bagvsv32'; check storage nodes, devices or pools
No available device on storage node bagvsv32
broker: no matching devices for save
I assume that your pool configuration is wrong (to 'narrow'). This might be especially due to the fast that you assigned devices to pools. Such is possible but it is not really necessary - as mentioned, it will narrow your pool selection criteria. The best example is in fact the pool Default which you are not allowed to change but it NW will still be able to use it. So avoid the device criteria whenever possible.
The other issue is that you have to be aware that loading/scanning and mounting are different issues. You can scan any compatible media in any appropriate device - but if you want to mount it (which you must do before you start a backup/recovery process), this will only be possible in a device which is allowed for the specific pool. Such would explain your specific behavior.
I assume it would have already be addressed by now?
As the client bagvsv32 being mentioned is also a storage node, does it have itself stated as storagenode to be used? As otherwise it would try to use the NW server as storagenode.
But as your error states "check storage nodes, devices or pools No available device on storage node bagvsv32" it seems to recognize to use itself as storage node might more likely be the pool in question not having option to use the devices fro this storage node. So are there restrictions in pool assignments? Are the devices from this storage node assigned to the pool in question?
Too bad also that the dreaded "check storage nodes, devices or pools" at times had nothing whatsoever to do with any of that, but was a rather generic error message that could be caused by completely unrelated other issues.
We tended to have some separation even way in the past when still using tape (now pretty much only datadomain via ddboost), to have OS backup go over the network via a shared storage node, while large database backups where using a specific san pool, that only dedicated storage nodes were supposed to use. This was at a tim when we were heavily sharing drives, hence would want to use them as effective as possible, hence send the much smaller OS backups over the LAN, while DB backups went over the SAN. We split up everything pretty much in san based pools and non-san pools and pools which were retention based, so 1wek, 2week, 3week, month, quarter and year pools and what not so that tapes would contain as much as possible active data without having to revert too much to staging to make tapes available again (which we still had to do... often)), restricting which device from which storage node, dedicated or shared, was supposed to use which pools.
Just to clarify - if you define a (remote) device, the client is automatically promoted to a storage node. And of course it will automatically be added as such in the configuration/resources.
smart4sugar
1 Rookie
•
8 Posts
1963
0
Posted March 11th, 2021 05:00
Thanks @barry_beckers for your detailed answer.
Meanwhile our problem has been solved.
First thing I've found, was that the tape storage node (bagsv32) was not listed under StorageNodeProperties -> Configuration -> Advanced Devices -> Clone storage nodes. After adding it there it was possible again to use the tape devices.
Second source of the trouble was a parallel installed networker client on the tape storage node (in addition with a different version). Since I have removed it, everything runs fine again.
Thanks to all the supporters in this thread for helping me!
Nico