The best way to see the logs and errors is to - check logs. Logs are usually in - logs directory. Main log file is on backup server and it is called daemon.raw. You can read it with nsr_render_log command and inside you will most likely find timestamps and what went wrong.
This issue arises whenever you try to label more than 20 tapes for the veriy first time. As the "Max. Consecutive Errors" counter is set to 20 NW will disable the device with the 21st media.
Best practice is to use Auto Media Management (AMM) for the jukebox.
If you do not want that, do not label more than 19 tapes in a row.
Good to know, thanks, but I don't think that's what happened... I had a backup running over the weekend, and it did about 6TB onto 2 LTO5 tapes, then failed to finish (~6TB more to go) because the tape device was disabled. No tapes were being labeled.
This very same error & disabled device happened once before, the same device and same backup. This backup is using some Imation LTO5 tapes, whereas I've always bought Quantum LTO4 & LTO4 tapes. I'm thinking (just a theory) that there were some write errors caused by the Imation media being just not quite as good as the Quantum, which disabled the tape drive; maybe a bad spot in one tape.
I would like to select out errors from the log associated with that device.
I wish Networker had a nice GUI interface to allow this, maybe that's in Networker 8? Or coming soon?
NetWorker log can be very big - to use GUI frontend to read that would be horrible - especially with java GUI. Just check daemon.raw and you will be able to trace what happened and when. Check that timestamp against system logs too.
Definitely *is* confusing (which is why a Useful, Friendly, GUI log viewer/filter would be great), but that's the output from the command: "nsr_render_log -l -F "/dev/rmt/1cbn" daemon.raw > tape_errs.txt" -- there aren't hardly any errors shown so why the device is disabled is not at all clear. The ordering is all from networker, I didn't do anything to affect the ordering. I'm puzzled by the lack of errors shown.
Cannot seem to get a date range select to work for nsr_render_log, it's rather arcane.
I'm slightly confused by your output as I see date on 3rd and then 2nd and then 3rd again (where that @ character is).
Anyway, tape did get mounted and some 12 hours later verification failed. From this point, we don't know what was happening with the volume so I would suggest to check the log for all entries between 6pm on 2nd and 6:30am on 3rd - with focus on volume called FullCIESIN-F-201303-02. Check also system logs for that period to see if any errors have occurred.
It is impossible for us to know what was the counter for errors at that point, was it already high and one error pushed it over error threshold or did all 20 happen with this particular run - that should be visible from the log. You should see from the log if volume was marked full prematurely or not.
Something like nsr_render_log -S '03/02/13 18:00:00' -E '03/03/13 06:30:00' /nsr/logs/daemon.raw > /tmp/foo does not work? I used that all the time (different OS though) and if works nice.
The key here is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT. Now we have to step back and look at bigger picture. You mentioned already that you have mix of LTO5 and LTO4 tapes. It looks as if this has been caused by LTO5. Now. the message you get, I suspect, is passed by CDI protocol (common device interface) to NetWorker. This is not NetWorker message, but something coming back either by driver (OS level) or library itself.
Yes, that does work, thank you. You've been very helpful. Silly me, I was trying to go by the documentation, e.g., from the 7.6SP2 command reference (which is why a handy GUI would be great, even if it were just a friendly front-end for nsr_render_log):
_render_log -S " l may 30 4:00 " <log_file_name>’
And, your command did find the errors. I don't understand why the -F didn't work, though, seems to me that it should have.
Errors below, I see them, but they don't make sense.... I.e., what caused the errors? Media problems? I had labeled the backup pool tapes on this very same device, iirc, certainly in the same library which only has 2 drives, only this one is selected for that media pool. The "WORM CAPABLE for device /dev/rmt/1cbn has been set" is puzzling
39074 03/03/13 05:24:18 0 0 1 3687 688 0 dbdev1 nsrjobd JOBS notice: Completed incremental database purge in 0 min 1 sec. Records purged: 0 38758 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn writing: Invalid argument, at file 423 record 20121 42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t ape FullCIESIN-F-201303-02 on /dev/rmt/1cbn is full 42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set 34353 03/03/13 06:06:09 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 34353 03/03/13 06:06:57 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:06:57 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE F ORMAT 42506 03/03/13 06:06:57 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set 34353 03/03/13 06:07:20 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 34353 03/03/13 06:08:12 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:08:12 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE F ORMAT 38758 03/03/13 06:09:55 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:11:02 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:12:10 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:12:10 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po sition FullCIESIN-F-201303-02 to file 423, record 20117 38758 03/03/13 06:12:37 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:13:23 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:14:30 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:14:30 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po sition FullCIESIN-F-201303-02 to file 423, record 20117 38758 03/03/13 06:14:57 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:15:43 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:16:51 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:16:51 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po sition FullCIESIN-F-201303-02 to file 423, record 20117 38758 03/03/13 06:17:18 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:18:04 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:19:12 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:19:12 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po sition FullCIESIN-F-201303-02 to file 423, record 20117 38758 03/03/13 06:19:38 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:20:25 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:21:32 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:21:32 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po sition FullCIESIN-F-201303-02 to file 423, record 20117 38758 03/03/13 06:21:59 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:22:46 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 38758 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT 42506 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd device disabled warning: Devi ce /dev/rmt/1cbn is automatically disabled. consecutive errors (21) exceeded the maximum consecutive errors allowed. Please fix the device or set a higher value for the Max consecutive errors attribute in the device resource. 38758 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd device disabled 2 warning: De vice /dev/rmt/1cbn is automatically disabled.
Further to what I said, CDI is picking up something that is standard signaling for errors described under ASC/ASCQ codes. You can find list of those on several places on the net, for example here:
They are OS independent so ignore Linux bit mentioned above in link. Some codes are general and some spaces are reserved for tape library vendor specific codes. The one you got, as per link above, is ASC 30 ASCQ 02.
NetWorker comes with program which gives you clue about some specific with ascdcode program:
# ascdcode
1673:ascdcode: usage: ascdcode [ -o vendor id [ -p product id ] ] asc ascq
It's an LTO5 tape drive, and that tape is LTO5. LTO5 tape drives can read & write LTO5 & LTO4 tapes. I've been using mostly LTO4 tapes in this library because of lower volume backups, but I've been using LTO5 tapes in another, identical libary with the same tape drives without problem. And, I've had several backups work with LTO5 tapes in this one. The drive should sense which kind of tape is in it, in any case. This isn't like SDLT where the SDLT tapes weren't compatible for writing across the generations SDLT/SDLT320, like the reference link in your post #10.
This backup had used 2 tapes, with 2700GB written to the first tape of the backup (...201303-01, it's marked Full) and 3300GB to the 2nd tape (-02, also now marked full). I still wonder if it isn't some strange media error, otherwise why would the drive sense "Incompatible Format" after writing 3300GB to that tape? Networker should be unloading the tape, since it's full, and loading the next one, it seems.
The tape device properties are set for SCSI commands, not CDI,. fwiw.
So odd, these Incompatible Format errors pop up just after the 2nd tape gets full, but the first tape works just fine (see the messages below)... I cannot help but wonder about the media and/or tape drive, but how can this be tested?
First tape gets filled up, ejected & 2nd tape (which later gets the errors in earlier post above) gets loaded, working as it should:
38758 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn writing: Invalid argument, at file 329 record 42363
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 tape FullCIESIN-F-201303-01 on /dev/rmt/1cbn is full
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:03 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:45 2 0 0 1 648 0 dbdev1 nsrd media info: verification of volume "FullCIESIN-F-201303-01", volid 2150682982 succeeded.
42506 03/02/13 18:02:45 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 tape FullCIESIN-F-201303-01 used 2748 GB of 1541 GB capacity
I've used 4 of these Imation LTO5 tapes on the other library, it did not have a problem filling them during a full backup.
I will check the firmware, that's good hygiene anyway.
Guess it's between the tape mfr. and the tape drive what's going on, not a Networker thing. Although I don't like the fact that the log renderer doesn't catch the "Incompatible Format" message when I select for that device (-F option), seems like a bug to me.
ble1
4 Operator
•
14.4K Posts
0
March 4th, 2013 07:00
The best way to see the logs and errors is to - check logs. Logs are usually in - logs directory. Main log file is on backup server and it is called daemon.raw. You can read it with nsr_render_log command and inside you will most likely find timestamps and what went wrong.
bingo.1
2.4K Posts
0
March 4th, 2013 07:00
This issue arises whenever you try to label more than 20 tapes for the veriy first time. As the "Max. Consecutive Errors" counter is set to 20 NW will disable the device with the 21st media.
Best practice is to use Auto Media Management (AMM) for the jukebox.
If you do not want that, do not label more than 19 tapes in a row.
dstrom
41 Posts
0
March 4th, 2013 08:00
Good to know, thanks, but I don't think that's what happened... I had a backup running over the weekend, and it did about 6TB onto 2 LTO5 tapes, then failed to finish (~6TB more to go) because the tape device was disabled. No tapes were being labeled.
This very same error & disabled device happened once before, the same device and same backup. This backup is using some Imation LTO5 tapes, whereas I've always bought Quantum LTO4 & LTO4 tapes. I'm thinking (just a theory) that there were some write errors caused by the Imation media being just not quite as good as the Quantum, which disabled the tape drive; maybe a bad spot in one tape.
I would like to select out errors from the log associated with that device.
I wish Networker had a nice GUI interface to allow this, maybe that's in Networker 8? Or coming soon?
ble1
4 Operator
•
14.4K Posts
0
March 4th, 2013 10:00
NetWorker log can be very big - to use GUI frontend to read that would be horrible - especially with java GUI. Just check daemon.raw and you will be able to trace what happened and when. Check that timestamp against system logs too.
dstrom
41 Posts
0
March 4th, 2013 11:00
A GUI would necessarily have to filter and/or page through the logs, could be done, if there was a will & budget$.
The logs don't show any errors for this tape device, so I still can't see what caused the disablement (service mode):
# nsr_render_log -l -F "/dev/rmt/1cbn" daemon.raw > tape_errs.txt
# view tape_errs.txt
"tape_errs.txt" [Read only] 242 lines, 28263 characters
MsgID TimeStamp Severity Category ErrorNo ThreadID ProcessID ActivityID HostName
ProgramName RenderedMessage
12312 01/25/13 13:21:57 2 0 0 1 11475 0 dbdev1 nsrd /dev/rmt/1cbn is now write
protected
67985 01/25/13 16:24:20 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `DEV-F-2012
12-03' from slot `3' into device `/dev/rmt/1cbn'.
38752 01/25/13 16:25:06 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
38752 01/25/13 16:25:20 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Label without m
ount operation in progress
38752 01/25/13 16:25:42 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 01/25/13 16:26:06 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `DEV-F-20
1301-06' from device `/dev/rmt/1cbn' to slot 3.
67985 01/25/13 16:26:40 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `DEV-F-2012
12-04' from slot `4' into device `/dev/rmt/1cbn'.
38752 01/25/13 16:27:22 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
38752 01/25/13 16:27:36 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Label without m
ount operation in progress
@
"tape_errs.txt" [Read only] 242 lines, 28263 characters
38752 03/02/13 10:02:31 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-01 on /dev/rmt/1cbn is full
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:03 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
38752 03/02/13 18:03:03 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/02/13 18:03:27 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIES
IN-F-201303-01' from device `/dev/rmt/1cbn' to slot 6.
67985 03/02/13 18:04:01 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `FullCIESIN
72504 03/01/13 11:07:35 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn verify label op
eration failed: Tape label read for volume ? in pool ?, is not recognised by Net
worker: I/O error
38752 03/01/13 11:07:36 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Label without m
ount operation in progress
38752 03/01/13 11:07:40 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/01/13 11:08:12 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIES
IN-F-201303-03' from device `/dev/rmt/1cbn' to slot 8.
67985 03/01/13 11:08:46 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `-' from sl
ot `9' into device `/dev/rmt/1cbn'.
38752 03/01/13 11:09:26 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
72504 03/01/13 11:09:41 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn verify label op
eration failed: Tape label read for volume ? in pool ?, is not recognised by Net
worker: I/O error
38752 03/01/13 11:09:41 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Label without m
ount operation in progress
38752 03/01/13 11:09:42 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/01/13 11:10:12 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIES
IN-F-201303-04' from device `/dev/rmt/1cbn' to slot 9.
@
38752 03/01/13 11:09:42 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/01/13 11:10:12 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIES
IN-F-201303-04' from device `/dev/rmt/1cbn' to slot 9.
67985 03/01/13 21:00:10 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `DEV-F-2013
02-03' from slot `3' into device `/dev/rmt/1cbn'.
38752 03/01/13 21:00:55 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
38752 03/01/13 21:01:08 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
38752 03/02/13 10:00:05 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/02/13 10:01:00 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `DEV-F-20
1302-03' from device `/dev/rmt/1cbn' to slot 3.
67985 03/02/13 10:01:33 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `FullCIESIN
-F-201303-01' from slot `6' into device `/dev/rmt/1cbn'.
38752 03/02/13 10:02:16 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
38752 03/02/13 10:02:31 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-01 on /dev/rmt/1cbn is full
@
38752 03/02/13 10:02:31 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-01 on /dev/rmt/1cbn is full
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:03 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
38752 03/02/13 18:03:03 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
67986 03/02/13 18:03:27 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIES
IN-F-201303-01' from device `/dev/rmt/1cbn' to slot 6.
67985 03/02/13 18:04:01 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `FullCIESIN
-F-201303-02' from slot `7' into device `/dev/rmt/1cbn'.
38752 03/02/13 18:04:41 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Verify label op
eration in progress
12312 03/02/13 18:04:56 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn is now enabled
read/write
38752 03/02/13 18:04:56 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-02 on /dev/rmt/1cbn is full
@
38752 03/02/13 18:04:56 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Mount operation
in progress
42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-02 on /dev/rmt/1cbn is full
42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
42506 03/03/13 06:06:57 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
42506 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd device disabled warning: Devi
ce /dev/rmt/1cbn is automatically disabled. consecutive errors (21) exceeded the
maximum consecutive errors allowed. Please fix the device or set a higher value
for the Max consecutive errors attribute in the device resource.
42506 03/03/13 06:24:20 2 0 0 1 648 0 dbdev1 nsrd media notice: Save set (79181
1224) afsisdata.ciesin.columbia.edu:/data2 volume FullCIESIN-F-201303-02 on /dev
/rmt/1cbn is being terminated because: Media verification failed
42506 03/03/13 06:24:20 2 0 0 1 648 0 dbdev1 nsrd media notice: Save set (80858
8440) afsisdata.ciesin.columbia.edu:/data1 volume FullCIESIN-F-201303-02 on /dev
/rmt/1cbn is being terminated because: Media verification failed
42506 03/03/13 06:24:20 2 0 0 1 648 0 dbdev1 nsrd media notice: Save set (10099
15023) afsisdata.ciesin.columbia.edu:/data3 volume FullCIESIN-F-201303-02 on /de
v/rmt/1cbn is being terminated because: Media verification failed
38752 03/03/13 06:24:21 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation
in progress
dstrom
41 Posts
0
March 4th, 2013 12:00
Definitely *is* confusing (which is why a Useful, Friendly, GUI log viewer/filter would be great), but that's the output from the command: "nsr_render_log -l -F "/dev/rmt/1cbn" daemon.raw > tape_errs.txt" -- there aren't hardly any errors shown so why the device is disabled is not at all clear. The ordering is all from networker, I didn't do anything to affect the ordering. I'm puzzled by the lack of errors shown.
Cannot seem to get a date range select to work for nsr_render_log, it's rather arcane.
ble1
4 Operator
•
14.4K Posts
0
March 4th, 2013 12:00
I'm slightly confused by your output as I see date on 3rd and then 2nd and then 3rd again (where that @ character is).
Anyway, tape did get mounted and some 12 hours later verification failed. From this point, we don't know what was happening with the volume so I would suggest to check the log for all entries between 6pm on 2nd and 6:30am on 3rd - with focus on volume called FullCIESIN-F-201303-02. Check also system logs for that period to see if any errors have occurred.
It is impossible for us to know what was the counter for errors at that point, was it already high and one error pushed it over error threshold or did all 20 happen with this particular run - that should be visible from the log. You should see from the log if volume was marked full prematurely or not.
ble1
4 Operator
•
14.4K Posts
1
March 4th, 2013 12:00
Something like nsr_render_log -S '03/02/13 18:00:00' -E '03/03/13 06:30:00' /nsr/logs/daemon.raw > /tmp/foo does not work? I used that all the time (different OS though) and if works nice.
ble1
4 Operator
•
14.4K Posts
0
March 5th, 2013 06:00
The key here is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT. Now we have to step back and look at bigger picture. You mentioned already that you have mix of LTO5 and LTO4 tapes. It looks as if this has been caused by LTO5. Now. the message you get, I suspect, is passed by CDI protocol (common device interface) to NetWorker. This is not NetWorker message, but something coming back either by driver (OS level) or library itself.
I suspect something similar to what is described here happens to you as well: Legato back-up solutions - Tape Device Media Type incorrect
With that in mind, do you know which drives do you use? You can verify that by running inquire -s from the storage node.
dstrom
41 Posts
0
March 5th, 2013 06:00
Yes, that does work, thank you. You've been very helpful. Silly me, I was trying to go by the documentation, e.g., from the 7.6SP2 command reference (which is why a handy GUI would be great, even if it were just a friendly front-end for nsr_render_log):
_render_log -S " l may 30 4:00 " <log_file_name>’
And, your command did find the errors. I don't understand why the -F didn't work, though, seems to me that it should have.
Errors below, I see them, but they don't make sense.... I.e., what caused the errors? Media problems? I had labeled the backup pool tapes on this very same device, iirc, certainly in the same library which only has 2 drives, only this one is selected for that media pool. The "WORM CAPABLE for device /dev/rmt/1cbn has been set" is puzzling
39074 03/03/13 05:24:18 0 0 1 3687 688 0 dbdev1 nsrjobd JOBS notice: Completed
incremental database purge in 0 min 1 sec. Records purged: 0
38758 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
writing: Invalid argument, at file 423 record 20121
42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 t
ape FullCIESIN-F-201303-02 on /dev/rmt/1cbn is full
42506 03/03/13 06:05:25 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
34353 03/03/13 06:06:09 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive
status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
34353 03/03/13 06:06:57 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive
status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:06:57 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE F
ORMAT
42506 03/03/13 06:06:57 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for
device /dev/rmt/1cbn has been set
34353 03/03/13 06:07:20 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive
status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
34353 03/03/13 06:08:12 2 0 0 1 788 0 dbdev1 nsrmmd tape_bsf bsf failed: drive
status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:08:12 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: tape_bsf bsf failed: drive status is CANNOT READ MEDIUM - INCOMPATIBLE F
ORMAT
38758 03/03/13 06:09:55 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:11:02 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:12:10 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:12:10 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po
sition FullCIESIN-F-201303-02 to file 423, record 20117
38758 03/03/13 06:12:37 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:13:23 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:14:30 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:14:30 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po
sition FullCIESIN-F-201303-02 to file 423, record 20117
38758 03/03/13 06:14:57 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:15:43 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:16:51 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:16:51 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po
sition FullCIESIN-F-201303-02 to file 423, record 20117
38758 03/03/13 06:17:18 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:18:04 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:19:12 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:19:12 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po
sition FullCIESIN-F-201303-02 to file 423, record 20117
38758 03/03/13 06:19:38 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:20:25 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:21:32 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:21:32 2 0 0 1 648 0 dbdev1 nsrd media emergency: could not po
sition FullCIESIN-F-201303-02 to file 423, record 20117
38758 03/03/13 06:21:59 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 421: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:22:46 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
38758 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn
moving: fsf 423: drive status is CANNOT READ MEDIUM - INCOMPATIBLE FORMAT
42506 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd device disabled warning: Devi
ce /dev/rmt/1cbn is automatically disabled. consecutive errors (21) exceeded the
maximum consecutive errors allowed. Please fix the device or set a higher value
for the Max consecutive errors attribute in the device resource.
38758 03/03/13 06:23:53 2 0 0 1 648 0 dbdev1 nsrd device disabled 2 warning: De
vice /dev/rmt/1cbn is automatically disabled.
ble1
4 Operator
•
14.4K Posts
0
March 5th, 2013 06:00
Further to what I said, CDI is picking up something that is standard signaling for errors described under ASC/ASCQ codes. You can find list of those on several places on the net, for example here:
The Linux SCSI programming HOWTO: Additional sense codes and additional sense code qualifiers
They are OS independent so ignore Linux bit mentioned above in link. Some codes are general and some spaces are reserved for tape library vendor specific codes. The one you got, as per link above, is ASC 30 ASCQ 02.
NetWorker comes with program which gives you clue about some specific with ascdcode program:
# ascdcode
1673:ascdcode: usage: ascdcode [ -o vendor id [ -p product id ] ] asc ascq
Your one seems to be general one:
# ascdcode 0x30 0x02
ASC/ASCQ(0x30/0x02)
Cannot Read Medium - Incompatible Format
dstrom
41 Posts
0
March 5th, 2013 07:00
It's an LTO5 tape drive, and that tape is LTO5. LTO5 tape drives can read & write LTO5 & LTO4 tapes. I've been using mostly LTO4 tapes in this library because of lower volume backups, but I've been using LTO5 tapes in another, identical libary with the same tape drives without problem. And, I've had several backups work with LTO5 tapes in this one. The drive should sense which kind of tape is in it, in any case. This isn't like SDLT where the SDLT tapes weren't compatible for writing across the generations SDLT/SDLT320, like the reference link in your post #10.
This backup had used 2 tapes, with 2700GB written to the first tape of the backup (...201303-01, it's marked Full) and 3300GB to the 2nd tape (-02, also now marked full). I still wonder if it isn't some strange media error, otherwise why would the drive sense "Incompatible Format" after writing 3300GB to that tape? Networker should be unloading the tape, since it's full, and loading the next one, it seems.
The tape device properties are set for SCSI commands, not CDI,. fwiw.
So odd, these Incompatible Format errors pop up just after the 2nd tape gets full, but the first tape works just fine (see the messages below)... I cannot help but wonder about the media and/or tape drive, but how can this be tested?
First tape gets filled up, ejected & 2nd tape (which later gets the errors in earlier post above) gets loaded, working as it should:
38758 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media warning: /dev/rmt/1cbn writing: Invalid argument, at file 329 record 42363
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 tape FullCIESIN-F-201303-01 on /dev/rmt/1cbn is full
42506 03/02/13 18:01:13 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:03 2 0 0 1 648 0 dbdev1 nsrd media info: WORM capable for device /dev/rmt/1cbn has been set
42506 03/02/13 18:02:45 2 0 0 1 648 0 dbdev1 nsrd media info: verification of volume "FullCIESIN-F-201303-01", volid 2150682982 succeeded.
42506 03/02/13 18:02:45 2 0 0 1 648 0 dbdev1 nsrd media notice: LTO Ultrium-5 tape FullCIESIN-F-201303-01 used 2748 GB of 1541 GB capacity
42506 03/02/13 18:03:01 2 0 0 1 648 0 dbdev1 nsrd write completion notice: Writing to volume FullCIESIN-F-201303-01 completed
0 03/02/13 18:03:01 2 0 0 1 648 0 dbdev1 nsrd Operation 142 started : Load volume `FullCIESIN-F-201303-02', volume id `2133905880'..
42502 03/02/13 18:03:01 2 0 0 1 648 0 dbdev1 nsrd media waiting event: Waiting for 1 writable volume(s) to backup pool 'FullCIESIN' tape(s) on dbdev1
38752 03/02/13 18:03:03 2 0 0 1 648 0 dbdev1 nsrd /dev/rmt/1cbn Eject operation in progress
67986 03/02/13 18:03:27 2 0 0 1 723 0 dbdev1 nsrmmgd Unloading volume `FullCIESIN-F-201303-01' from device `/dev/rmt/1cbn' to slot 6.
67985 03/02/13 18:04:01 2 0 0 1 723 0 dbdev1 nsrmmgd Loading volume `FullCIESIN-F-201303-02' from slot `7' into device `/dev/rmt/1cbn'.
ble1
4 Operator
•
14.4K Posts
0
March 5th, 2013 08:00
CDI or not, it doesn't matter - NW will sense/pass ASC/ASCQ codes.
One question, you use same media and drives on second library? Do drives have same firmware?
Of course, it could be much simpler reality like - broken tape batch (though mostly those were the things of the past).
dstrom
41 Posts
0
March 5th, 2013 11:00
I've used 4 of these Imation LTO5 tapes on the other library, it did not have a problem filling them during a full backup.
I will check the firmware, that's good hygiene anyway.
Guess it's between the tape mfr. and the tape drive what's going on, not a Networker thing. Although I don't like the fact that the log renderer doesn't catch the "Incompatible Format" message when I select for that device (-F option), seems like a bug to me.
Thanks for the help.