This backup server (MKZSB001) is attached to a HP MSL6030 tape
library. We are running Legato Networker Administrator version 7.0. This
backup server is dedicated to backup our Exchange server via the
public LAN (not sure why it was not configured with a private LAN
initially). Only 1 tape drive is in use.
Background of the issue:
Usually the Exchange OS & DB full backup is run at 8:00pm and 9:00pm
daily. The OS backup usually finishes not later than 8:45pm, and the
Exchange DB not later than 1:00am. However, the backup on Sunday
2006-05-07 took very much longer than expected (actually on most Sundays
but it usually finishes by 5am). OS backup only finished at 3:41am, and
the DB backup never completed on time. As explained, MS Exchange has a
behaviour: If more than 1024 log files are written during the backup
process, the associated DB is dismounted. At around 9:00am on Monday
2006-05-08, the Exchange DB was dismounted due to this issue. The DBs
were able to be mounted, but a downtime of 5 hours was experienced as
the DB has to replay all log files since the previous night (I think
there were more than 2000 log files!).
I had checked on the Exchange server and with the network side, there is
nothing to indicate why the backup took such a long time. Therefore, I
need your assistance to assist in analyzing the Legato logs to find out
any reason/root cause.
First of all I hope you don't have really 7.0 version otherwise you have much bigger problem (in case you do, schedule upgrade with priority 1 for your own sake).
There is not much you can do right now. Here is the story; Legato in its handling relays completely on system and its health. When backup fails, unless there is a application bug, you can be certain there is a problem with system somewhere.
In your case you have 1 drive dedicated to backup of Exchange node. From reading your post it is not clear if you use different pools for file system and Exchange data, but I assume you do not (otherwise pool used for file system would block the tape from pool used for Exchange).
From what you say it seems like both file system and Exchange did perform badly in performance (when pushing the stream to storage node). If that is the case it indicates that was not an issue with Exchange application. Sometimes you will see bad and errorless performance when you are dealing with drive which is about to die (or needs cleaning) or bad media. I don't think that applies to you, but to verify you need to check: - average speed of writing on each tape during this backups (can be done via daemon.log) for both OS and EXCH (pay attention to number of streams while doing analysis too) - did the performance increased to normal afterwards for the same client (after the incident)
Last question above is important as you need to determine at which point things got to normal and for what reason. For example, if everything did come back to its senses backup without application restart I can hardly related this problem to NW. Second, were any other backups slower at that time (going to any other storage nodes)?
Given that you didn't see anything in, I assume event logs on the box didn't show any extra error/warning than usual. I assume same is done with network switch (if not, do it - usually network guys do have monitoring). What about storage node itself? Any indication of trouble in the logs there?
From NW side the only log you can check really is daemon.log now - and check if this started to be an issue since the backup started or during the during (perhaps streams were doing fine and then they slowed down at certain point.
There are few more things of course, but if this used to work before than we can skip them now. One way of going back is to check if everything works fine now and if it does what has been done/changed. Then, go from that point further. If nothing was changed then this could be one time glitch where reason for bad performance remains unknown.
Thanks for your advice, here is the part daemon.log. From the log is that anything wrong, as i know the backup slow at previous Sunday only, last sunday seem like every thing OK, do you thing is Legato problem?
Here is the Log: 05/07/06 01:06:48 nsrd: mkzse201:MSEXCH:IS/SG2MB done saving to pool 'Daily' (MSX008) 323 GB 05/07/06 01:07:01 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX008) 05/07/06 01:08:06 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX008) 333 MB 05/07/06 01:08:16 nsrd: savegroup notice: MSX_Daily completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/07/06 01:08:49 nsrd: write completion notice: Writing to volume MSX008 complete 05/07/06 20:00:01 nsrd: savegroup info: starting OS_Daily (with 2 client(s)) 05/07/06 20:00:01 nsrd: savegroup info: starting OS_Monthly (with 2 client(s)) 05/07/06 20:00:27 nsrd: media notice: Jukebox 'MSL6000' only has 32 GB available on 1 volume(s) 05/07/06 20:00:27 nsrd: media info: suggest mounting MSX823 on mkzsb001.ap.infineon.com for writing to pool 'Monthly' 05/07/06 20:00:27 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:00:28 nsrd: Jukebox 'MSL6000' failed: server is busy, please retry request 05/07/06 20:00:29 nsrmmd #2: Start nsrmmd #2, with PID 3816, at HOST mkzsb001.ap.infineon.com 05/07/06 20:00:37 nsrd: \\.\Tape1 Eject operation in progress 05/07/06 20:00:57 nsrd: media info: Suggest manually labeling a new writable volume for pool 'Daily' 05/07/06 20:00:57 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:01:01 nsrmmd #3: Start nsrmmd #3, with PID 4324, at HOST mkzsb001.ap.infineon.com 05/07/06 20:01:20 nsrd: media info: loading volume MSX823 into \\.\Tape1 05/07/06 20:01:36 nsrd: \\.\Tape1 Verify label operation in progress 05/07/06 20:01:57 nsrd: \\.\Tape1 Mount operation in progress 05/07/06 20:02:08 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:02:39 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:03:30 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:03:30 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Monthly' (MSX823) 05/07/06 20:04:11 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Monthly' (MSX823) 75 MB 05/07/06 20:04:37 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Monthly' (MSX823) 05/07/06 20:04:38 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Monthly' (MSX823) 2789 KB 05/07/06 20:05:02 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Monthly' (MSX823) 05/07/06 20:05:02 nsrmmdbd: media db is saving its data. This may take a while. 05/07/06 20:05:03 nsrmmdbd: media db is open for business. 05/07/06 20:05:03 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Monthly' (MSX823) 2191 KB 05/07/06 20:05:26 nsrd: savegroup notice: OS_Monthly completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/07/06 20:05:44 nsrmmdbd: Starting compression of media database 05/07/06 20:05:46 nsrd: write completion notice: Writing to volume MSX823 complete 05/07/06 20:05:47 nsrmmdbd: Finished compression of media database 05/07/06 20:05:47 nsrd: index notice: nsrim has finished cross checking the media db 05/07/06 20:05:51 nsrd: media info: suggest mounting MSX008 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/07/06 20:05:52 nsrd: \\.\Tape1 Eject operation in progress 05/07/06 20:07:41 nsrd: media info: loading volume MSX008 into \\.\Tape1 05/07/06 20:07:58 nsrd: \\.\Tape1 Verify label operation in progress 05/07/06 20:08:19 nsrd: \\.\Tape1 Mount operation in progress 05/07/06 20:09:08 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/07/06 20:09:08 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Daily' (MSX008) 05/07/06 20:09:08 nsrd: mkzse201:SYSTEM STATE:\ saving to pool 'Daily' (MSX008) 05/07/06 20:09:08 nsrd: mkzse201:SYSTEM DB:\ saving to pool 'Daily' (MSX008) 05/07/06 20:09:08 nsrd: mkzse201:SYSTEM FILES:\ saving to pool 'Daily' (MSX008) 05/07/06 20:09:48 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Daily' (MSX008) 75 MB 05/07/06 20:10:03 nsrd: mkzse201:C:\ saving to pool 'Daily' (MSX008) 05/07/06 20:11:32 nsrd: mkzse201:SYSTEM STATE:\ done saving to pool 'Daily' (MSX008) 18 MB 05/07/06 20:11:40 nsrd: mkzse201:SYSTEM DB:\ done saving to pool 'Daily' (MSX008) 1112 KB 05/07/06 20:11:44 nsrd: mkzse201:E:\ saving to pool 'Daily' (MSX008) 05/07/06 20:11:44 nsrd: mkzse201:D:\ saving to pool 'Daily' (MSX008) 05/07/06 20:18:12 nsrd: mkzse201:D:\ done saving to pool 'Daily' (MSX008) 5235 MB 05/07/06 20:18:28 nsrd: mkzse201:F:\ saving to pool 'Daily' (MSX008) 05/07/06 20:20:04 nsrd: mkzse201:F:\ done saving to pool 'Daily' (MSX008) 1057 MB 05/07/06 20:20:29 nsrd: mkzse201:G:\ saving to pool 'Daily' (MSX008) 05/07/06 20:22:36 nsrd: mkzse201:G:\ done saving to pool 'Daily' (MSX008) 1317 MB 05/07/06 20:23:00 nsrd: mkzse201:H:\ saving to pool 'Daily' (MSX008) 05/07/06 20:23:08 nsrd: mkzse201:H:\ done saving to pool 'Daily' (MSX008) 101 MB 05/07/06 20:23:27 nsrd: mkzse201:M:\ saving to pool 'Daily' (MSX008) 05/07/06 20:23:30 nsrd: mkzse201:M:\ done saving to pool 'Daily' (MSX008) 1 KB 05/07/06 20:29:42 nsrd: mkzse201:SYSTEM FILES:\ done saving to pool 'Daily' (MSX008) 245 MB 05/07/06 21:00:00 nsrd: savegroup info: starting MSX_Monthly (with 1 client(s)) 05/07/06 21:00:00 nsrd: savegroup info: starting MSX_Daily (with 1 client(s)) 05/07/06 21:00:00 nsrd: savegroup notice: MSX_Monthly completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/07/06 21:01:32 nsrd: mkzse201:MSEXCH: saving to pool 'Daily' (MSX008) 05/08/06 00:53:08 nsrd: media warning: \\.\Tape1 writing: The physical end of the tape has been reached., at file 292 record 14718 05/08/06 00:53:08 nsrd: media notice: LTO Ultrium tape MSX008 on \\.\Tape1 is full 05/08/06 00:53:08 nsrd: media notice: LTO Ultrium tape MSX008 used 303 GB of 200 GB capacity 05/08/06 00:55:57 nsrd: media info: verification of volume "MSX008", volid 73185567 succeeded. 05/08/06 00:56:13 nsrd: write completion notice: Writing to volume MSX008 complete 05/08/06 00:56:13 nsrd: media notice: Jukebox 'MSL6000' only has 32 GB available on 1 volume(s) 05/08/06 00:56:13 nsrd: media info: suggest relabeling MSX009 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/08/06 00:56:13 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 00:56:13 nsrd: nsrjb notice: nsrjb -j MSL6000 -O67 -l -R -M MSX009 05/08/06 00:56:13 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 00:56:53 nsrd: media info: loading volume MSX009 into \\.\Tape1 05/08/06 00:57:08 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 00:57:29 nsrd: \\.\Tape1 Label without mount operation in progress 05/08/06 00:57:30 nsrd: media info: LTO Ultrium tape MSX009 will be over-written 05/08/06 00:57:30 nsrd: deleted media notice: Deleted volume: volid=506623189, volname=MSX009, location=MSL6000 05/08/06 00:57:38 nsrd: \\.\Tape1 is now enabled 05/08/06 00:57:38 nsrd: \\.\Tape1 Mount operation in progress 05/08/06 00:57:54 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 00:57:54 nsrd: mkzse201:MSEXCH: saving to pool 'Daily' (MSX009) 26 GB 05/08/06 00:57:54 nsrd: mkzse201:C:\ saving to pool 'Daily' (MSX009) 7331 MB 05/08/06 00:57:54 nsrd: mkzse201:E:\ saving to pool 'Daily' (MSX009) 29 GB 05/08/06 01:53:42 nsrd: mkzse201:MSEXCH: done saving to pool 'Daily' (MSX009) 33 GB 05/08/06 01:54:13 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX009) 05/08/06 03:30:25 nsrd: mkzse201:E:\ done saving to pool 'Daily' (MSX009) 47 GB 05/08/06 03:38:58 nsrd: mkzse201:C:\ done saving to pool 'Daily' (MSX009) 15 GB 05/08/06 03:39:12 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX009) 05/08/06 03:40:21 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX009) 340 MB 05/08/06 03:40:27 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Daily' (MSX009) 05/08/06 03:40:30 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Daily' (MSX009) 2748 KB 05/08/06 03:40:52 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Daily' (MSX009) 05/08/06 03:40:53 nsrmmdbd: media db is saving its data. This may take a while. 05/08/06 03:40:53 nsrmmdbd: media db is open for business. 05/08/06 03:40:56 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Daily' (MSX009) 2197 KB 05/08/06 03:41:16 nsrd: savegroup notice: OS_Daily completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/08/06 08:56:05 nsrd: mkzse201:MSEXCH:IS/SG2MB done saving to pool 'Daily' (MSX009) 107 GB 05/08/06 08:56:20 savegrp: mkzse201:MSEXCH: will retry 1 more time(s) 05/08/06 08:56:49 nsrd: write completion notice: Writing to volume MSX009 complete 05/08/06 08:57:44 nsrd: mkzse201:MSEXCH: saving to pool 'Daily' (MSX009) 05/08/06 09:02:36 savegrp: group MSX_Daily aborted. 05/08/06 09:02:36 savegrp: killing pid 4220 05/08/06 09:02:36 nsrd: savegroup alert: MSX_Daily aborted, total 1 client(s), 0 Hostname(s) Unresolved, 1 Failed, 0 Succeeded. (mkzse201 Failed) * mkzse201:MSEXCH: nsrxchsv_ese.exe Version: 3.1.0.90_QuickFix_LGTpa44858 (LGTpa44858) Supporting Exchange 2000 * mkzse201:MSEXCH: System Version: 5.0 Build 2195 Service Pack 4 * mkzse201:MSEXCH: ESEBCLI2.DLL Version: 6.0.6603.0 Service Pack 4 * mkzse201:MSEXCH: Performing an Exchange save operation; save level is Full (Legato Backup Level: full). * mkzse201:MSEXCH: Backup storage group SG1PF to mkzsb001.ap.infineon.com * mkzse201:MSEXCH: Backup database: SG1PFStore (MKZSE201) {5210DABF-DE0A-4130-9287-4622B2E5D5A6} * mkzse201:MSEXCH: Backup of SG1PFStore (MKZSE201) was successful * mkzse201:MSEXCH: Backup database: SG1VipMail (MKZSE201) {E083943F-4163-4F21-BF6B-AE1C62470F89} * mkzse201:MSEXCH: Backup of SG1VipMail (MKZSE201) was successful * mkzse201:MSEXCH: Backup of logs for storage group SG1PF was successful * mkzse201:MSEXCH: Completed backup of SG1PF Status: 0x0 * mkzse201:MSEXCH: The operation completed successfully. * mkzse201:MSEXCH: Backup storage group SG2MB to mkzsb001.ap.infineon.com * mkzse201:MSEXCH: Backup database: SG2FabMail1 (MKZSE201) {FBAF8638-DD6F-4078-82BF-DDB7DC517A18} * mkzse201:MSEXCH: Backup of SG2FabMail1 (MKZSE201) was successful * mkzse201:MSEXCH: Backup database: SG2StdMail1 (MKZSE201) {A0AF8E4F-0E64-44DE-82B1-3198F2E34A09} * mkzse201:MSEXCH: HrESEBackupReadFile: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: HrESEBackupCloseFile: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: Backup database: SG2StdMail2 (MKZSE201) {2F45ED64-2B7B-400C-8475-4A42F069FA6E} * mkzse201:MSEXCH: backup_db_guid can't get size for H:\exchsrvr\SG2MB\SG2StdMail2.edb * mkzse201:MSEXCH: backup_db_guid can't get size for H:\exchsrvr\SG2MB\SG2StdMail2.stm * mkzse201:MSEXCH: HrESEBackupOpenFile: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: Backup database: SG2StdMail3 (MKZSE201) {4C63F76E-F169-443C-BB59-D1286E9B1196} * mkzse201:MSEXCH: backup_db_guid can't get size for E:\exchsrvr\SG2MB\SG2StdMail3.edb * mkzse201:MSEXCH: backup_db_guid can't get size for E:\exchsrvr\SG2MB\SG2StdMail3.stm * mkzse201:MSEXCH: HrESEBackupOpenFile: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: Backup database: SG2StdMail4 (MKZSE201) {08F7A7A5-9EEC-44BE-BBD8-4C6EEB3ED66F} * mkzse201:MSEXCH: backup_db_guid can't get size for E:\exchsrvr\SG2MB\SG2StdMail4.edb * mkzse201:MSEXCH: backup_db_guid can't get size for E:\exchsrvr\SG2MB\SG2StdMail4.stm * mkzse201:MSEXCH: HrESEBackupOpenFile: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: HrESEBackupGetLogAndPatchFiles: Error returned from an ESE function call (d). * mkzse201:MSEXCH: * mkzse201:MSEXCH: hrErrorFromESECall last error: -1090 (0xfffffbbe) * mkzse201:MSEXCH: Completed backup of SG2MB Status: 0x4de * mkzse201:MSEXCH: Continue with work in progress. * mkzse201:MSEXCH: G:\rt_2002_1Q\nsr\db_apps\aime\svmain.c(1117): process_is_backup() = 0x000004de mkzse201: MSEXCH:IS level=Full (Legato Backup Level: full), 140 GB 11:54:56 18 file(s) * mkzse201:MSEXCH: Backup operation finished with errors. 05/08/06 09:02:36 nsrd: runq: NSR group MSX_Daily exited with return code 1. 05/08/06 09:02:36 nsrd: mkzse201:MSEXCH: done saving to pool 'Daily' (MSX009) 6999 MB 05/08/06 09:03:13 nsrd: write completion notice: Writing to volume MSX009 complete 05/08/06 11:22:10 nsrd: \\.\Tape1 unmounted MSX009 05/08/06 11:23:28 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:23:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:23:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:23:48 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:23:48 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:23:49 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:24:41 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:25:02 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:25:02 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:25:02 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:25:02 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:25:02 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:25:54 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:26:10 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:26:10 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:26:10 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:26:10 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:26:10 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:27:18 nsrd: nsrjb notice: nsrjb -j MSL6000 -Y -O70 -L -g -bMonthly -S 30-30 05/08/06 11:27:33 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:27:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:27:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:27:48 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:27:48 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:27:49 nsrd: \\.\Tape1 Label without mount operation in progress 05/08/06 11:27:49 nsrd: media info: LTO Ultrium tape will be over-written 05/08/06 11:27:52 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:28:55 nsrd: nsrjb notice: nsrjb -j MSL6000 -Y -O71 -L -g -bDaily -S 26-27 05/08/06 11:29:11 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:29:26 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:29:26 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:29:26 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:29:26 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:29:27 nsrd: \\.\Tape1 Label without mount operation in progress 05/08/06 11:29:27 nsrd: media info: LTO Ultrium tape will be over-written 05/08/06 11:29:30 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:30:28 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 11:30:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:30:48 nsrmmd #1: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:30:48 nsrd: media warning: \\.\Tape1 moving: tape_bsf bsf failed: drive status is The end of data was reached 05/08/06 11:30:48 nsrd: media warning: \\.\Tape1 reading: no tape label found 05/08/06 11:30:49 nsrd: \\.\Tape1 Label without mount operation in progress 05/08/06 11:30:49 nsrd: media info: LTO Ultrium tape will be over-written 05/08/06 11:30:52 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 11:49:48 nsrd: \\.\Tape1 unmounted (already) 05/08/06 16:10:01 nsrd: savegroup info: starting OS_Daily (with 2 client(s)) 05/08/06 16:10:01 nsrd: savegroup info: starting OS_Monthly (with 2 client(s)) 05/08/06 16:10:27 nsrd: media info: suggest mounting MSX823 on mkzsb001.ap.infineon.com for writing to pool 'Monthly' 05/08/06 16:10:27 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/08/06 16:10:28 nsrd: media info: loading volume MSX823 into \\.\Tape1 05/08/06 16:10:29 nsrmmd #2: Start nsrmmd #2, with PID 4640, at HOST mkzsb001.ap.infineon.com 05/08/06 16:10:44 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 16:11:05 nsrd: \\.\Tape1 Mount operation in progress 05/08/06 16:12:23 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/08/06 16:12:23 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Monthly' (MSX823) 05/08/06 16:13:02 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Monthly' (MSX823) 75 MB 05/08/06 16:13:22 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Monthly' (MSX823) 05/08/06 16:13:23 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Monthly' (MSX823) 2798 KB 05/08/06 16:13:47 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Monthly' (MSX823) 05/08/06 16:13:48 nsrmmdbd: media db is saving its data. This may take a while. 05/08/06 16:13:48 nsrmmdbd: media db is open for business. 05/08/06 16:13:48 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Monthly' (MSX823) 2201 KB 05/08/06 16:14:12 nsrd: savegroup notice: OS_Monthly completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/08/06 16:14:27 nsrd: write completion notice: Writing to volume MSX823 complete 05/08/06 16:14:31 nsrd: media info: suggest mounting MSX009 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/08/06 16:14:31 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 16:14:32 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 16:16:20 nsrd: media info: loading volume MSX009 into \\.\Tape1 05/08/06 16:16:37 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 16:16:58 nsrd: \\.\Tape1 Mount operation in progress 05/08/06 16:18:53 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 16:18:53 nsrd: mkzse201:SYSTEM STATE:\ saving to pool 'Daily' (MSX009) 05/08/06 16:18:53 nsrd: mkzse201:SYSTEM DB:\ saving to pool 'Daily' (MSX009) 05/08/06 16:18:53 nsrd: mkzse201:C:\ saving to pool 'Daily' (MSX009) 05/08/06 16:18:53 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Daily' (MSX009) 05/08/06 16:19:33 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Daily' (MSX009) 75 MB 05/08/06 16:19:44 nsrd: mkzse201:D:\ saving to pool 'Daily' (MSX009) 05/08/06 16:19:50 nsrd: mkzse201:SYSTEM DB:\ done saving to pool 'Daily' (MSX009) 1112 KB 05/08/06 16:20:12 nsrd: mkzse201:SYSTEM STATE:\ done saving to pool 'Daily' (MSX009) 18 MB 05/08/06 16:20:13 nsrd: mkzse201:E:\ saving to pool 'Daily' (MSX009) 05/08/06 16:20:29 nsrd: mkzse201:F:\ saving to pool 'Daily' (MSX009) 05/08/06 16:20:35 nsrd: mkzse201:F:\ done saving to pool 'Daily' (MSX009) 05/08/06 16:20:54 nsrd: mkzse201:G:\ saving to pool 'Daily' (MSX009) 05/08/06 16:21:03 nsrd: mkzse201:G:\ done saving to pool 'Daily' (MSX009) 3 KB 05/08/06 16:21:19 nsrd: mkzse201:H:\ saving to pool 'Daily' (MSX009) 05/08/06 16:21:24 nsrd: mkzse201:H:\ done saving to pool 'Daily' (MSX009) 15 KB 05/08/06 16:21:44 nsrd: mkzse201:M:\ saving to pool 'Daily' (MSX009) 05/08/06 16:21:46 nsrd: mkzse201:M:\ done saving to pool 'Daily' (MSX009) 05/08/06 16:22:09 nsrd: mkzse201:E:\ done saving to pool 'Daily' (MSX009) 8891 KB 05/08/06 16:35:48 nsrd: mkzse201:D:\ done saving to pool 'Daily' (MSX009) 15 GB 05/08/06 16:36:55 nsrd: mkzse201:C:\ done saving to pool 'Daily' (MSX009) 543 MB 05/08/06 16:37:09 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX009) 05/08/06 16:38:09 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX009) 340 MB 05/08/06 16:38:24 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Daily' (MSX009) 05/08/06 16:38:25 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Daily' (MSX009) 2838 KB 05/08/06 16:38:49 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Daily' (MSX009) 05/08/06 16:38:49 nsrmmdbd: media db is saving its data. This may take a while. 05/08/06 16:38:50 nsrmmdbd: media db is open for business. 05/08/06 16:38:50 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Daily' (MSX009) 2206 KB 05/08/06 16:39:13 nsrd: savegroup notice: OS_Daily completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/08/06 16:39:35 nsrd: write completion notice: Writing to volume MSX009 complete 05/08/06 17:00:01 nsrd: savegroup info: starting MSX_Daily (with 1 client(s)) 05/08/06 17:00:01 nsrd: savegroup info: starting MSX_Monthly (with 1 client(s)) 05/08/06 17:00:01 nsrd: savegroup notice: MSX_Monthly completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/08/06 17:01:07 nsrd: mkzse201:MSEXCH: saving to pool 'Daily' (MSX009) 05/08/06 18:55:16 nsrd: mkzse201:MSEXCH: done saving to pool 'Daily' (MSX009) 34 GB 05/08/06 18:55:27 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX009) 05/08/06 20:12:39 nsrd: media warning: \\.\Tape1 writing: The physical end of the tape has been reached., at file 301 record 5278 05/08/06 20:12:39 nsrd: media notice: LTO Ultrium tape MSX009 on \\.\Tape1 is full 05/08/06 20:12:39 nsrd: media notice: LTO Ultrium tape MSX009 used 308 GB of 200 GB capacity 05/08/06 20:13:53 nsrd: media info: verification of volume "MSX009", volid 4083033851 succeeded. 05/08/06 20:14:09 nsrd: write completion notice: Writing to volume MSX009 complete 05/08/06 20:14:09 nsrd: media info: suggest mounting MSX041 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/08/06 20:14:09 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 20:14:10 nsrd: Jukebox 'MSL6000' failed: server is busy, please retry request 05/08/06 20:14:11 nsrmmd #2: Start nsrmmd #2, with PID 4492, at HOST mkzsb001.ap.infineon.com 05/08/06 20:14:19 nsrd: \\.\Tape1 Eject operation in progress 05/08/06 20:15:01 nsrd: media info: loading volume MSX041 into \\.\Tape1 05/08/06 20:15:18 nsrd: \\.\Tape1 Verify label operation in progress 05/08/06 20:15:42 nsrd: \\.\Tape1 is now enabled 05/08/06 20:15:42 nsrd: \\.\Tape1 Mount operation in progress 05/08/06 20:15:44 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/08/06 20:15:44 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX041) 108 GB 05/08/06 22:37:45 nsrd: mkzse201:MSEXCH:IS/SG2MB done saving to pool 'Daily' (MSX041) 325 GB 05/08/06 22:37:53 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX041) 05/08/06 22:39:01 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX041) 340 MB 05/08/06 22:39:08 nsrd: savegroup notice: MSX_Daily completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/08/06 22:39:25 nsrmmdbd: Starting compression of media database 05/08/06 22:39:28 nsrmmdbd: Finished compression of media database 05/08/06 22:39:28 nsrd: index notice: nsrim has finished cross checking the media db 05/08/06 22:39:43 nsrd: write completion notice: Writing to volume MSX041 complete 05/09/06 18:00:01 nsrd: savegroup info: starting OS_Daily (with 2 client(s)) 05/09/06 18:00:01 nsrd: savegroup info: starting OS_Monthly (with 2 client(s)) 05/09/06 18:01:56 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Daily' (MSX041) 05/09/06 18:01:56 nsrd: mkzse201:SYSTEM DB:\ saving to pool 'Daily' (MSX041) 05/09/06 18:01:56 nsrd: mkzse201:SYSTEM STATE:\ saving to pool 'Daily' (MSX041) 05/09/06 18:01:56 nsrd: mkzse201:C:\ saving to pool 'Daily' (MSX041) 05/09/06 18:02:40 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Daily' (MSX041) 75 MB 05/09/06 18:02:57 nsrd: mkzse201:SYSTEM DB:\ done saving to pool 'Daily' (MSX041) 1112 KB 05/09/06 18:02:59 nsrd: mkzse201:D:\ saving to pool 'Daily' (MSX041) 05/09/06 18:03:10 nsrd: mkzse201:SYSTEM STATE:\ done saving to pool 'Daily' (MSX041) 18 MB 05/09/06 18:03:24 nsrd: mkzse201:E:\ saving to pool 'Daily' (MSX041) 05/09/06 18:03:24 nsrd: mkzse201:F:\ saving to pool 'Daily' (MSX041) 05/09/06 18:03:31 nsrd: mkzse201:F:\ done saving to pool 'Daily' (MSX041) 05/09/06 18:03:48 nsrd: mkzse201:G:\ saving to pool 'Daily' (MSX041) 05/09/06 18:03:57 nsrd: mkzse201:G:\ done saving to pool 'Daily' (MSX041) 3 KB 05/09/06 18:04:13 nsrd: mkzse201:H:\ saving to pool 'Daily' (MSX041) 05/09/06 18:04:19 nsrd: mkzse201:H:\ done saving to pool 'Daily' (MSX041) 15 KB 05/09/06 18:04:46 nsrd: mkzse201:M:\ saving to pool 'Daily' (MSX041) 05/09/06 18:04:48 nsrd: mkzse201:E:\ done saving to pool 'Daily' (MSX041) 78 MB 05/09/06 18:04:48 nsrd: mkzse201:M:\ done saving to pool 'Daily' (MSX041) 05/09/06 18:21:05 nsrd: mkzse201:C:\ done saving to pool 'Daily' (MSX041) 850 MB 05/09/06 18:24:04 nsrd: mkzse201:D:\ done saving to pool 'Daily' (MSX041) 25 GB 05/09/06 18:24:41 nsrd: write completion notice: Writing to volume MSX041 complete 05/09/06 18:26:00 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX041) 05/09/06 18:27:24 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX041) 339 MB 05/09/06 18:27:34 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Daily' (MSX041) 05/09/06 18:27:36 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Daily' (MSX041) 2757 KB 05/09/06 18:27:59 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Daily' (MSX041) 05/09/06 18:28:00 nsrmmdbd: media db is saving its data. This may take a while. 05/09/06 18:28:00 nsrmmdbd: media db is open for business. 05/09/06 18:28:01 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Daily' (MSX041) 2228 KB 05/09/06 18:28:24 nsrd: savegroup notice: OS_Daily completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/09/06 18:28:35 nsrd: write completion notice: Writing to volume MSX041 complete 05/09/06 18:30:12 nsrd: media info: suggest mounting MSX823 on mkzsb001.ap.infineon.com for writing to pool 'Monthly' 05/09/06 18:30:12 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/09/06 18:30:12 nsrd: Jukebox 'MSL6000' failed: server is busy, please retry request 05/09/06 18:30:22 nsrmmd #2: Start nsrmmd #2, with PID 4500, at HOST mkzsb001.ap.infineon.com 05/09/06 18:30:32 nsrd: \\.\Tape1 Eject operation in progress 05/09/06 18:31:54 nsrd: media info: loading volume MSX823 into \\.\Tape1 05/09/06 18:32:09 nsrd: \\.\Tape1 Verify label operation in progress 05/09/06 18:32:30 nsrd: \\.\Tape1 Mount operation in progress 05/09/06 18:33:46 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Monthly' tape(s) on mkzsb001.ap.infineon.com 05/09/06 18:33:46 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato saving to pool 'Monthly' (MSX823) 05/09/06 18:34:30 nsrd: mkzsb001.ap.infineon.com:c:\Program Files\Legato done saving to pool 'Monthly' (MSX823) 75 MB 05/09/06 18:34:39 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com saving to pool 'Monthly' (MSX823) 05/09/06 18:34:41 nsrd: mkzsb001.ap.infineon.com:index:mkzsb001.ap.infineon.com done saving to pool 'Monthly' (MSX823) 2797 KB 05/09/06 18:35:04 nsrd: mkzsb001.ap.infineon.com:bootstrap saving to pool 'Monthly' (MSX823) 05/09/06 18:35:05 nsrmmdbd: media db is saving its data. This may take a while. 05/09/06 18:35:05 nsrmmdbd: media db is open for business. 05/09/06 18:35:06 nsrd: mkzsb001.ap.infineon.com:bootstrap done saving to pool 'Monthly' (MSX823) 2229 KB 05/09/06 18:35:29 nsrd: savegroup notice: OS_Monthly completed, total 2 client(s), 0 Hostname(s) Unresolved, 0 Failed, 2 Succeeded. 05/09/06 18:35:52 nsrd: write completion notice: Writing to volume MSX823 complete 05/09/06 19:00:01 nsrd: savegroup info: starting MSX_Monthly (with 1 client(s)) 05/09/06 19:00:01 nsrd: savegroup info: starting MSX_Daily (with 1 client(s)) 05/09/06 19:00:01 nsrd: savegroup notice: MSX_Monthly completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/09/06 19:00:34 nsrd: media info: suggest mounting MSX041 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/09/06 19:00:34 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 19:00:35 nsrd: Jukebox 'MSL6000' failed: server is busy, please retry request 05/09/06 19:00:36 nsrmmd #2: Start nsrmmd #2, with PID 4184, at HOST mkzsb001.ap.infineon.com 05/09/06 19:00:44 nsrd: \\.\Tape1 Eject operation in progress 05/09/06 19:02:34 nsrd: media info: loading volume MSX041 into \\.\Tape1 05/09/06 19:02:49 nsrd: \\.\Tape1 Verify label operation in progress 05/09/06 19:03:10 nsrd: \\.\Tape1 Mount operation in progress 05/09/06 19:04:20 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 19:04:20 nsrd: mkzse201:MSEXCH: saving to pool 'Daily' (MSX041) 05/09/06 19:31:40 nsrd: mkzse201:MSEXCH: done saving to pool 'Daily' (MSX041) 34 GB 05/09/06 19:31:59 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX041) 05/09/06 19:48:30 nsrd: media warning: \\.\Tape1 writing: The physical end of the tape has been reached., at file 295 record 15067 05/09/06 19:48:30 nsrd: media notice: LTO Ultrium tape MSX041 on \\.\Tape1 is full 05/09/06 19:48:30 nsrd: media notice: LTO Ultrium tape MSX041 used 303 GB of 200 GB capacity 05/09/06 19:51:22 nsrd: media info: verification of volume "MSX041", volid 3965631255 succeeded. 05/09/06 19:51:38 nsrd: write completion notice: Writing to volume MSX041 complete 05/09/06 19:51:38 nsrd: media info: suggest mounting MSX042 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/09/06 19:51:38 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 19:51:38 nsrd: \\.\Tape1 Eject operation in progress 05/09/06 19:52:18 nsrd: media info: loading volume MSX042 into \\.\Tape1 05/09/06 19:52:33 nsrd: \\.\Tape1 Verify label operation in progress 05/09/06 19:52:57 nsrd: \\.\Tape1 is now enabled 05/09/06 19:52:57 nsrd: \\.\Tape1 Mount operation in progress 05/09/06 19:53:28 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 19:53:28 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX042) 22 GB 05/09/06 22:55:39 nsrd: media warning: \\.\Tape1 writing: The physical end of the tape has been reached., at file 283 record 15055 05/09/06 22:55:39 nsrd: media notice: LTO Ultrium tape MSX042 on \\.\Tape1 is full 05/09/06 22:55:39 nsrd: media notice: LTO Ultrium tape MSX042 used 295 GB of 200 GB capacity 05/09/06 22:58:28 nsrd: media info: verification of volume "MSX042", volid 3948854121 succeeded. 05/09/06 22:58:44 nsrd: write completion notice: Writing to volume MSX042 complete 05/09/06 22:58:44 nsrd: media info: suggest relabeling MSX040 on mkzsb001.ap.infineon.com for writing to pool 'Daily' 05/09/06 22:58:44 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 22:58:44 nsrd: nsrjb notice: nsrjb -j MSL6000 -O83 -l -R -M MSX040 05/09/06 22:58:45 nsrd: \\.\Tape1 Eject operation in progress 05/09/06 22:59:23 nsrd: media info: loading volume MSX040 into \\.\Tape1 05/09/06 22:59:38 nsrd: \\.\Tape1 Verify label operation in progress 05/09/06 22:59:59 nsrd: \\.\Tape1 Label without mount operation in progress 05/09/06 23:00:00 nsrd: media info: LTO Ultrium tape MSX040 will be over-written 05/09/06 23:00:00 nsrd: deleted media notice: Deleted volume: volid=171168945, volname=MSX040, location=MSL6000 05/09/06 23:00:11 nsrd: \\.\Tape1 is now enabled 05/09/06 23:00:11 nsrd: \\.\Tape1 Mount operation in progress 05/09/06 23:00:24 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Daily' tape(s) on mkzsb001.ap.infineon.com 05/09/06 23:00:25 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX040) 317 GB 05/09/06 23:04:54 nsrd: mkzse201:MSEXCH:IS/SG2MB done saving to pool 'Daily' (MSX040) 324 GB 05/09/06 23:05:19 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 saving to pool 'Daily' (MSX040) 05/09/06 23:06:19 nsrd: mkzsb001.ap.infineon.com:index:mkzse201 done saving to pool 'Daily' (MSX040) 339 MB 05/09/06 23:06:33 nsrd: savegroup notice: MSX_Daily completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succeeded. 05/09/06 23:06:51 nsrmmdbd: Starting compression of media database 05/09/06 23:06:54 nsrd: write completion notice: Writing to volume MSX040 complete 05/09/06 23:06:54 nsrmmdbd: Finished compression of media database 05/09/06 23:06:54 nsrd: index notice: nsrim has finished cross checking the media db
You also using old module too I believe you should do some refresh to your applications.
As for backup, I can see when it comes to file system that system partition was quite slow.. you can see that when you check timings for C drive and SYSTEM savesets (DB and FILES). This may indicate box was busy with "something" (or simply run out of resources at that point).
Exchange part is strange as it shows: 05/08/06 01:53:42 nsrd: mkzse201:MSEXCH: done saving to pool 'Daily' (MSX009) 33 GB 05/08/06 01:54:13 nsrd: mkzse201:MSEXCH:IS/SG2MB saving to pool 'Daily' (MSX009)
05/08/06 08:56:05 nsrd: mkzse201:MSEXCH:IS/SG2MB done saving to pool 'Daily' (MSX009) 107 GB 05/08/06 08:56:20 savegrp: mkzse201:MSEXCH: will retry 1 more time(s)
What is the saveset you use? IS should be enough - if you have MSEXCH you can skip storage groups (even many users will prefer storage group backups only and once a month full MSEXCH backup).
Also, it is highly recommended not to have file system backup and module one running at the same time or overlapping. You may wish to consider to reschedule file system backup to be far away enough from DB backup.
I can't say based on supplied input. Do the things work fine now? If yes was there any application restart?
Based on what I see from log, I would say something did happen on that machine triggering slower performance output toward storage node. I can't say what, but I would not hold NW as primary suspect here (I believe backup performance was purely consequence of something else in this case).
But did you ask customer if they restarted anything (you need to rule out that as possible cause for "back to life" state).
7.0 is no longer supported. At the time when it was it was extremely buggy release which could lead you to problems with restore (so soon after 7.0.1 was released). Currently supported versions are 7.1.x (7.1.4 is current), 7.2.x (7.2.2 is current) and 7.3. You probably wish to go to 7.2.2 right now.
You may also consider to inspect newer features in 4.0.x and 4.1.x module for Exchange.
ble1
6 Operator
•
14354 Posts
•
56186 Points
436
0
Posted May 17th, 2006 22:00
First of all I hope you don't have really 7.0 version otherwise you have much bigger problem (in case you do, schedule upgrade with priority 1 for your own sake).
There is not much you can do right now. Here is the story; Legato in its handling relays completely on system and its health. When backup fails, unless there is a application bug, you can be certain there is a problem with system somewhere.
In your case you have 1 drive dedicated to backup of Exchange node. From reading your post it is not clear if you use different pools for file system and Exchange data, but I assume you do not (otherwise pool used for file system would block the tape from pool used for Exchange).
From what you say it seems like both file system and Exchange did perform badly in performance (when pushing the stream to storage node). If that is the case it indicates that was not an issue with Exchange application. Sometimes you will see bad and errorless performance when you are dealing with drive which is about to die (or needs cleaning) or bad media. I don't think that applies to you, but to verify you need to check:
- average speed of writing on each tape during this backups (can be done via daemon.log) for both OS and EXCH (pay attention to number of streams while doing analysis too)
- did the performance increased to normal afterwards for the same client (after the incident)
Last question above is important as you need to determine at which point things got to normal and for what reason. For example, if everything did come back to its senses backup without application restart I can hardly related this problem to NW. Second, were any other backups slower at that time (going to any other storage nodes)?
Given that you didn't see anything in, I assume event logs on the box didn't show any extra error/warning than usual. I assume same is done with network switch (if not, do it - usually network guys do have monitoring). What about storage node itself? Any indication of trouble in the logs there?
From NW side the only log you can check really is daemon.log now - and check if this started to be an issue since the backup started or during the during (perhaps streams were doing fine and then they slowed down at certain point.
There are few more things of course, but if this used to work before than we can skip them now. One way of going back is to check if everything works fine now and if it does what has been done/changed. Then, go from that point further. If nothing was changed then this could be one time glitch where reason for bad performance remains unknown.