Problems with SL3000 and LTO-4 drives

Question

Hi all,

We have recently installed new backup environment with linux storage nodes and Sun SL3000 library with LTO-4 drives. My problems are frequent mount/unmount errors which will cause drives to be in service mode.

Storage node is 64 bit Suse linux enterprise 10 and NW version is 7.4.4.

Any thoughts? We already increased unload and load sleeps but no luck. When you reboot the drive from SLConsole or switch power off/on from drive it will unload without errors and will work until next error will occur.

We have another storage node on IBM P6 environment and no problems there.

Here are the error logs from NW and from Linux.

Networker:

0 09/07/09 00:00:15 2 0 0 2834257808 19396 0 penw01 nsrd Operation 4522 started : Load volume `PE00
17', volume id `1315491865'..
0 09/07/09 00:00:17 2 0 0 1190508432 19537 0 penw01 nsrmmgd Volume `PE0017' will be loaded in devic
e `rd=b-penwstgn01:/dev/nst1' with tag `D:001000'
38752 09/07/09 00:00:44 2 0 0 2834257808 19396 0 penw01 nsrd rd=b-penwstgn01:/dev/nst1 Verify label
operation in progress
38758 09/07/09 00:04:55 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 opening: Input/output error
38758 09/07/09 00:04:55 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 reading: read open error, Input/output error
7224 09/07/09 00:04:56 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4522]. Expected volume `PE0017' in slot `17'. The actual volume is ` '.
38752 09/07/09 00:04:56 2 0 0 2834257808 19396 0 penw01 nsrd rd=b-penwstgn01:/dev/nst1 Eject operat
ion in progress
38758 09/07/09 00:09:08 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 opening: Input/output error
38758 09/07/09 00:09:08 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 moving: eject: Bad file descriptor
7224 09/07/09 00:09:09 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4522]. eject: Bad file descriptor
7224 09/07/09 00:09:09 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4522]. Expected volume `PE0017' in slot `17'. The actual volume is ` '.
12361 09/07/09 00:09:13 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', ope
ration # 4522]. Finished with status: failed
0 09/07/09 00:09:23 2 0 0 2834257808 19396 0 penw01 nsrd Operation 4523 started : Load volume `PE00
39', volume id `4118325597'..
38752 09/07/09 00:09:25 2 0 0 2834257808 19396 0 penw01 nsrd rd=b-penwstgn01:/dev/nst1 Eject operat
ion in progress
38758 09/07/09 00:13:36 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 opening: Input/output error
38758 09/07/09 00:13:36 2 0 0 2834257808 19396 0 penw01 nsrd media warning: rd=b-penwstgn01:/dev/ns
t1 moving: eject: Bad file descriptor
7224 09/07/09 00:13:36 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4523]. eject: Bad file descriptor
0 09/07/09 00:13:36 2 0 0 1190508432 19537 0 penw01 nsrmmgd Volume `PE0017' is unloaded from device
`rd=b-penwstgn01:/dev/nst1' with tag `D:001000'
0 09/07/09 00:13:53 2 0 0 1190508432 19537 0 penw01 nsrmmgd lcpd 1 at host penwstgn01 reported erro
r 'Jukebox:rd=b-penwstgn01:SL3000 access:scsidev@2.1.0 failed:MOVE MEDIUM key:5 status:CHECK CONDITI
ON UNKNOWN, Medium Not Present' for the command `5'.
7224 09/07/09 00:13:53 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4523]. Jukebox:rd=b-penwstgn01:SL3000 access:scsidev@2.1.0 failed:MOVE MEDIUM key:5 status:C
HECK CONDITION UNKNOWN, Medium Not Present
4690 09/07/09 00:13:53 2 0 0 1190508432 19537 0 penw01 nsrmmgd Jukebox:rd=b-penwstgn01:SL3000 acces
s:scsidev@2.1.0 failed:MOVE MEDIUM key:5 status:CHECK CONDITION UNKNOWN, Medium Not Present
7224 09/07/09 00:13:53 2 0 0 2834257808 19396 0 penw01 nsrd [Jukebox `rd=b-penwstgn01:SL3000', oper
ation # 4523]. eject: Bad file descriptor

Linux:

Sep 7 00:13:52 penwstgn01 kernel: sg_cmd_done: Current: sense key: Illegal Request
Sep 7 00:13:52 penwstgn01 kernel: Additional sense: Medium not present
Sep 7 00:18:26 penwstgn01 kernel: sg_cmd_done: Current: sense key: Illegal Request
Sep 7 00:18:26 penwstgn01 kernel: Additional sense: Medium not present
Sep 7 00:23:57 penwstgn01 sshd[26948]: Accepted keyboard-interactive/pam for patrol from 192.168.0.109 port 2324 ssh2
Sep 7 00:24:19 penwstgn01 kernel: sg_cmd_done: Current: sense key: Illegal Request
Sep 7 00:24:19 penwstgn01 kernel: Additional sense: Medium not present
Sep 7 00:29:05 penwstgn01 kernel: sg_cmd_done: Current: sense key: Illegal Request
Sep 7 00:29:05 penwstgn01 kernel: Additional sense: Medium not present
Sep 7 00:34:19 penwstgn01 kernel: sg_cmd_done: Current: sense key: Illegal Request
Sep 7 00:34:19 penwstgn01 kernel: Additional sense: Medium not present

DavidHampson · Answer

Do you have the latest drivers installed for your tape devices?

jsallila · Answer

We are using generic Linux drivers. Should we use IBM's drivers? I checked IBM's site and there is new 1.27 driver published last month.

ble1 · Answer

I didn't check now, but I clearly remember IBM drivers under Linux were not supported before. I played a bit with those on HPUX and those drivers are really dangerous so if you can stay with native ones that would be great.

The error as seen here may suggest following:
a) your device order is incorrect
b) if it is not a) then loaded device does not respond which may indicate SAN/SCSI issue

Most likely this is issue a).

jkernagh · Answer

I would agree with Hrvoje; your device order is definitely out.

The Linux OS log clearly states the sense key reporting 'Medium not present' and NetWorker is telling us it's getting back an unexpected medium: 'Expected volume x got NULL / y'. That means NetWorker has loaded for example Drive 1 which it thinks is nst0; whereas behind the scenes Drive 1 is now using nst5 due to an OS name change.

Linux kernels 2.6 and above support udev, which supports persistent naming. This will allow you to permanently fix device handles to physical drives, even in the event of a temporary or permanent device loss from anything other than the highest handle number (which persistent binding will not).

After you successfully do this you'll instead get handles like .../tape/by-id/...
Run inquire -p on the Linux host to see the new handles and confirm udev is correctly set up.
Then, rescan for new devices with the 'Use Persistent Names' box ticked, and you'll see the new devices added as unconfigured orange wrenches.
Last, 'Reconfigure Library', untick the old /dev/nstX devices, and tick instead the new /tape/by-id/ handles and your drive ordering issues should be a thing of the past.

A caveat: SL3000 may have the same DWWN feature as an SL8500; if so, before you set up udev and reconfigure your library ensure WWN and SN masking are enabled; this feature will mean you can swap tape drives upon failure with no need to rezone OR reconfigure NetWorker as the WWN and SN for each drive bay will permanently use the same masquerade.

I would also agree that IBM Atape drivers are not recommended for Linux and I'm pretty sure they're not supported either. Use st and I think you'll have much better results in general.

A good idea for Linux devices in NetWorker is to enable Simple Reservations (in the Advanced tab I think) which may help to prevent conflict as well.

HTH, James.

jsallila · Answer

This problem is now corrected. Problem was bug in the LTO-drive firmware. Bug affects only IBM LTO-4 drives and it has something to do with compression.

Firmware was downgraded and seems like library is working properly.

Message was edited by: JPS. Corrected information.
JPS

NetWorker

Problems with SL3000 and LTO-4 drives

Was this post helpful?