UNSOLVED
Networker Automatic Cloning Failing
Networker server 7.4.4 running on a Windows Server 2003 OS. Automatic cloning has failed for 26 savesets. Backups are done to disk, then cloned off to tape.
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 2416828894 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 2702040597 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 3540899834 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 3406682154 during savegroup operation has failed!
.......
We are also getting a number of these messages in the rendered daemon.log file:
media emergency: Bad or missing record in save set 2383274562, lost 17179 bytes starting at offset 28029933176.
However we don't get the message above for each saveset that failed to clone.
Looking at the daemon log found the following entry:-
38758 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media warning: E:\NSRDisk\LUN03 writing: No space left on device, at file 2383274562 record 866863 42506 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media notice: file disk Disk-LUN03 on E:\NSRDisk\LUN03 is full 42506 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media notice: file disk Disk-LUN03 used 452 GB of 1100 GB capacity 42506 15/05/2009 23:27:08 2 0 0 2144 4584 0 nsrd write completion notice: Writing to volume Disk-LUN03 complete
As you can see it failed to write to the Disk-LUN03 beyond 452 GB on the 15th May at 23:27, when each LUN has 1100 GB of available space.
This is just one example, multiple LUNs are sometimes not filling to capacity according to Networker.
Something happened to stop Networker writing the saveset to the disk. Not all of the data was saved and some was lost before it could continue on to the next volume, therefore the saveset has become corrupt and hence the cloning is failing. I suspect manual cloning off these savesets would fail as well as the savesets are corrupted.
This is happening for different savesets for different clients, and for different LUNs. Also automatic cloning doesn't always fail, this happens intermittently. This is a new disk library and tape library Scalar 50, and there is no evidence of SCSI or FC failures in the event logs on the server.
Can you suggest a resolution?
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 2416828894 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 2702040597 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 3540899834 during savegroup operation has failed!
0 16/05/2009 20:20:19 2 0 0 6204 4748 0 savegrp 05/16/09 20:20:19 savegrp: Automatic cloning of saveset 3406682154 during savegroup operation has failed!
.......
We are also getting a number of these messages in the rendered daemon.log file:
media emergency: Bad or missing record in save set 2383274562, lost 17179 bytes starting at offset 28029933176.
However we don't get the message above for each saveset that failed to clone.
Looking at the daemon log found the following entry:-
38758 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media warning: E:\NSRDisk\LUN03 writing: No space left on device, at file 2383274562 record 866863 42506 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media notice: file disk Disk-LUN03 on E:\NSRDisk\LUN03 is full 42506 15/05/2009 23:27:07 2 0 0 2144 4584 0 nsrd media notice: file disk Disk-LUN03 used 452 GB of 1100 GB capacity 42506 15/05/2009 23:27:08 2 0 0 2144 4584 0 nsrd write completion notice: Writing to volume Disk-LUN03 complete
As you can see it failed to write to the Disk-LUN03 beyond 452 GB on the 15th May at 23:27, when each LUN has 1100 GB of available space.
This is just one example, multiple LUNs are sometimes not filling to capacity according to Networker.
Something happened to stop Networker writing the saveset to the disk. Not all of the data was saved and some was lost before it could continue on to the next volume, therefore the saveset has become corrupt and hence the cloning is failing. I suspect manual cloning off these savesets would fail as well as the savesets are corrupted.
This is happening for different savesets for different clients, and for different LUNs. Also automatic cloning doesn't always fail, this happens intermittently. This is a new disk library and tape library Scalar 50, and there is no evidence of SCSI or FC failures in the event logs on the server.
Can you suggest a resolution?
