Troubleshooting Tape Library Drive Ordering Problems in NetWorker

In a Plug 'N Play operating system, devices are assigned SCSI target addresses in the order of discovery.

Since the order of discovery on a SAN is not fixed, and because connectivity loss causes Plug 'N play to redefine the target number, target numbers are not fixed.

Since Plug 'N Play systems name (and rename) devices based on enumeration order (including SCSI target ID) - any sort of accidental or intentional interruption in the operating system's connection to the device potentially results in renamed devices.

A 'drive ordering' problem describes a condition where NetWorker's configured driver name for a device does not match the actual name. This is most commonly a result of the driver name changing in the Operating System after initial NetWorker library configuration. This is typically a Plug 'N Play operating system issue, affecting Windows and Linux.
 
There are many errors and conditions associated with this problem, including, but not limited to:

• Error: 'nsrd: media info: failed unloading drive `{driver handle}' to slot {slot number}, error '69''
• Error: '{hostname} the destination component full'
• Error: '{driver handle} read open error, no such device or address'
• Error: 'opening: I/O error'
• Error: 'nsrd: Jukebox '{jukebox}' failed: expected volume '{volid}' got {volid}'
• Error: 'nsrd: Jukebox '{jukebox}' failed: expected volume '(volume_name)' got 'NULL''
• Error: 'read open error, device not ready'
• Error: 'nsrjb: Jukebox error, All allocated drives are not usable, unrecoverable operation errors'
• Error: 'nsrd: Jukebox '{jukebox}' failed: expected volume '{volid}' got {volid}'
• Error: 'nsrd: Jukebox '{jukebox}' failed: expected volume '{volume}' got 'NULL''
• Error: 'read open error, device not ready'
• Error: 'nsrjb: Jukebox error, All allocated drives are not usable, unrecoverable operation errors'
• Error: 'nsrd: media warning: {driver handle} reading: read open error: No media in drive.'
• Error: 'inventory: Bar code label `{barcode}' does not match media db bar code label, updating media db'
• Error: 'Illegal request, medium not present'
• Error: 'nsrd: media info: failed unloading drive `{driver handle}' to slot {slot number}'

At the time a Tape Library is initially configured in NetWorker, the library object is created associating the hard drives with their OS-generated driver handles they have at that moment in time. It is a static association which reflects the relationship at the time of configuration. For example, a library may have two devices:
Physical drive 1 = \\.\Tape0 (or perhaps /dev/nst0 in Linux)
Physical drive 2 =\\.\Tape1 (or /dev/nst1)

In a Plug 'n Play operating system like Windows or Linux, any sort of disappearance of the device from the system may cause the operating system to rename the devices; this includes simply rebooting the host, devices, or any of the connectivity hardware involved in the connection. Especially on a SAN, where device discovery may be disordered, the devices may be named differently on the next reboot, for example, as contrasting the above:
Physical drive 1 = \\.\Tape1  or /dev/nst1
Physical drive 2 =  \\.\Tape0  or /dev/nst0

Commands to these devices may still work, assuming there is any device using the wanted name. However, since the association of driver handle and physical element is no longer correct in the NetWorker Library element, the result of this is that NetWorker no longer knows the correct name of the devices, as they are now changed. For example, NetWorker may load a tape cartridge into a drive element, but use the original (and now, following the operating system event, incorrect) tape driver name to perform tape operations - i.e it may load tape drive 1, but issue commands to drive 2 (who has been reassigned drive 1's old name). This can result in a wide range of errors, assuming an unexpected volume (or none at all) is found. There are many possible causes of drive ordering conditions:

• Manual misconfiguration of library using jbconfig or jbedit commands
• Reboot of host, storage adapter, storage connectivity hardware or tape devices
• Temporary loss of connectivity to a device
• Disabling and reenabling the device in the operating system
• Operating system updates
• Device or storage adapter driver updates

Persistent Naming:

This is considered to be best practice and may be recommended by support even if you are not experiencing issues to proactively protect you. Please use the information from the following articles:

• Implementing Tape Device Name Persistence for Windows
• Implementing Tape Device Name Persistence for Linux