Full backup is performing more then once

Question

Hello,

I have a strange behaviour.
I want to backup a windows 2003 server (storage node). I create 2 clients:
client X: saveset ALL directive skip E:\Data\Actual Projects
client Y: saveset E:\Data\Actual Projects

Client X is running without any problem.
Client Y acts very strange. After all the writing he runs again with the message "No full backups of this save set were found in the media database; performing a ful
l backup"

See logging below. All the logging is from the group I started manually at 15:59.
From 23:00 you see the logging of client Y.

I'm running Networker 7.1.3 on Solaris 9

Best regards,

Carlo

08/08/06 15:59:47 nsrd: savegroup info: starting Gent_BigData (with 1 client(s))
08/08/06 15:59:55 nsrd: savegroup info: gntinfra1:E:\Data\Actual Projects: No full backups of this save set were found i
n the media database; performing a full backup
08/08/06 16:00:46 nsrd: gntinfra1:E:\Data\Actual Projects saving to pool 'GENT' (Gent.023)
08/08/06 16:04:04 nsrd: warning: pool `GENTmonth' changed while server is busy.
08/08/06 16:04:15 nsrd: warning: pool `EDG' changed while server is busy.
08/08/06 17:20:13 nsrd: gntinfra1:E:\Data\Actual Projects done saving to pool 'GENT' (Gent.023) 97 GB
08/08/06 17:26:38 nsrd: write completion notice: Writing to volume Gent.023 complete
gntinfra1:E:\Data\Actual Projects: No full backups of this save set were found in the media database; performing a ful
l backup
* gntinfra1:E:\Data\Actual Projects save: Diagnostic: This client is not licensed for VSS.
* gntinfra1:E:\Data\Actual Projects save: Diagnostic: E:\Data\Actual Projects will be saved without a snapshot being tak
en.
08/08/06 18:08:47 savegrp: gntinfra1:E:\Data\Actual Projects will retry 1 more time(s)
08/08/06 18:08:47 nsrd: savegroup info: gntinfra1:E:\Data\Actual Projects: No full backups of this save set were found i
n the media database; performing a full backup
08/08/06 18:11:31 nsrd: gntinfra1:E:\Data\Actual Projects saving to pool 'GENT' (Gent.023)
08/08/06 19:22:52 nsrd: gntinfra1:E:\Data\Actual Projects done saving to pool 'GENT' (Gent.023) 97 GB
08/08/06 19:29:08 nsrd: write completion notice: Writing to volume Gent.023 complete
* gntinfra1:E:\Data\Actual Projects 1 retry attempted
gntinfra1:E:\Data\Actual Projects: No full backups of this save set were found in the media database; performing a ful
l backup
* gntinfra1:E:\Data\Actual Projects save: Diagnostic: This client is not licensed for VSS.
* gntinfra1:E:\Data\Actual Projects save: Diagnostic: E:\Data\Actual Projects will be saved without a snapshot being tak
en.
08/08/06 20:17:32 savegrp: gntinfra1:E:\Data\Actual Projects will retry 0 more time(s)
08/08/06 20:17:32 savegrp: gntinfra1:E:\Data\Actual Projects will retry 0 more time(s)
08/08/06 20:17:32 nsrd: edgbckprd01:index:gntinfra1 saving to pool 'EDG' (EDG009)
08/08/06 20:17:32 nsrd: edgbckprd01:index:edgoeewpl saving to pool 'EDG' (EDG007)
08/08/06 20:17:34 nsrd: edgbckprd01:index:gntinfra1 done saving to pool 'EDG' (EDG009) 7659 KB
08/08/06 20:17:34 nsrd: savegroup alert: Gent_BigData completed, total 1 client(s), 0 Hostname(s) Unresolved, 1 Failed,
0 Succeeded. (gntinfra1 Failed)
08/08/06 20:17:34 nsrd: runq: NSR group Gent_BigData exited with return code 1.

08/08/06 23:01:32 nsrd: media info: suggest mounting Gent.023 on gntinfra1 for writing to pool 'GENT'
08/08/06 23:01:32 nsrd: media waiting event: Waiting for 1 writable volumes to backup pool 'GENT' tape(s) on gntinfra1
08/08/06 23:01:37 nsrd: rd=gntinfra1:\\.\Tape0 Verify label operation in progress
08/08/06 23:02:45 nsrd: rd=gntinfra1:\\.\Tape0 Mount operation in progress
08/08/06 23:03:35 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'GENT' tape(s) on gntinfra1
08/08/06 23:03:35 nsrd: gntinfra1:SYSTEM STATE:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:04:46 nsrd: gntinfra1:SYSTEM STATE:\ done saving to pool 'GENT' (Gent.023) 135 MB
08/08/06 23:05:00 nsrd: gntinfra1:SYSTEM DB:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:05:11 nsrd: gntinfra1:SYSTEM DB:\ done saving to pool 'GENT' (Gent.023) 10 MB
08/08/06 23:05:16 nsrd: gntinfra1:ASR:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:05:35 nsrd: gntinfra1:ASR:\ done saving to pool 'GENT' (Gent.023) 1269 KB
08/08/06 23:05:39 nsrd: gntinfra1:C:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:06:18 nsrd: gntinfra1:C:\ done saving to pool 'GENT' (Gent.023) 27 MB
08/08/06 23:06:24 nsrd: gntinfra1:D:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:06:25 nsrd: gntinfra1:D:\ done saving to pool 'GENT' (Gent.023)
08/08/06 23:06:33 nsrd: gntinfra1:E:\ saving to pool 'GENT' (Gent.023)
08/08/06 23:07:29 nsrd: gntinfra1:E:\ done saving to pool 'GENT' (Gent.023) 627 MB
08/08/06 23:07:30 nsrd: edgbckprd01:index:gntinfra1 saving to pool 'EDG' (EDG010)
08/08/06 23:07:33 nsrd: edgbckprd01:index:gntinfra1 done saving to pool 'EDG' (EDG010) 8107 KB
08/08/06 23:07:33 nsrd: savegroup notice: Gent completed, total 1 client(s), 0 Hostname(s) Unresolved, 0 Failed, 1 Succe
eded.
08/08/06 23:08:02 nsrd: write completion notice: Writing to volume Gent.023 complete

nsrhti · Answer

OK I 'll test it. It is a normal NTFS-filesystem.
I see also the following messages. But I don't know if this has something to do with this post.

08/08/06 18:08:07 nsrd: Aborting invalid connection from 0.0.0.0/19907 to 0.0.0.0/0., Connection timed out
08/08/06 18:08:17 nsrd: Aborting invalid connection from 0.0.0.0/26926 to 0.0.0.0/0., Connection timed out

This messages appears randomized duering differentvbackups.

ble1 · Answer

Strange indeed. Anything special about that directory like DFS or something else? What happens if you change saveset from E:\Data\Actual Projects to E:/Data/Actual Projects?

nsrhti · Answer

Ok I have changed the saveset, but the problem is the same.
This is what happens:
When he has done writing the saveset (90GB) I see the message
"08/09/06 15:19:57 nsrd: write completion notice: Writing to volume Gent.003 complete"
Normally the index of edginfra1 should be written to the backupserver.
But for a half hour there is no action, but I see the backup is still active.
Then I have the following message:
08/09/06 15:47:51 nsrd: Aborting invalid connection from 0.0.0.0/22520 to 0.0.0.0/0., Connection timed out
08/09/06 15:47:51 nsrd: Aborting invalid connection from 0.0.0.0/13709 to 0.0.0.0/0., Connection timed out

After this message he starts again the backup of the saveset.
So I think it is a connection problem, because the client is a storagenode at a remote site. I think there is a problem for writing the index of the client to the server.
But for the other policy (clientX - saveset ALL) there is no problem.

ble1 · Answer

You should open this with support. Error you see should say something like 'there is a communication problem/error between hostname1/IP1 and hostname2/IP2'. As it seems you have a problem in communication between two daemons on server itself (nsrd and nsrindexd?). That may indicate index issue so I would suggest to run nsrck -L6 edginfra1 and after that nsrck -m (to be safe on that front too). Then try it again (backup). If it fails, try manual (client initiated) save - that one should work. That try to save index for only that client and see if it run ok. Variation on the same subject would be to check 'no index save' on group where this backup is running and to see if it is successful then. If yes, then there is an issue with index probably, but the question remains why only after that saveset runs. Let's leave that for support or someone else to figure out (as it requires additional testing I won't go into right now).

nsrhti · Answer

OK I'll run the nsrck this evening.

I also have checked the firewall logging. Between the remote and local site there is a firewall. The default policy is to allow everything (therefore you have a firewall :-)

. Normally this should not be an issue.
But I see some strange messages:
"TCP packet out of state; First packet isn't SYN"
According to me this normal behaviour of a firewall when a TCP connection is established without any SYN-flag. But this is the reason why I have the message "Aborting Invalid connection..."

If I have the same problem after I run nsrck I will open a case at support.
Thanks for your input. You are really helpfull.

ble1 · Answer

Oh firewall... They always tend to add extra headache... if possible, add one more test, disable firewall and run the backup. Just so that you can say - firewall should not be an issue (or perhaps it should be - depends on test results).

nsrhti · Answer

First of all I will open a case at support.
It is not a firewall problem.

I also did a nsrck and the backup runs without a problem BUT
He did an incremental backup. But how can he do a incremental if all the full backups aren't succesfull.
If I start manually a full backup it fails, when I start an incremental it is succesfull.

Here some output:
root@edgbckprd01 / # mminfo -s edgbckprd01 -q "client=gntinfra1,name=E:\Data\Actual Projects" -ot -v
volume client date time size ssid fl lvl name
Gent.023 gntinfra1 08/08/06 15:58:32 97 GB 4107835661 cb full E:\Data\Actual Projects
Gent.023 gntinfra1 08/08/06 18:07:17 97 GB 4091066291 cb full E:\Data\Actual Projects
Gent.018 gntinfra1 08/10/06 09:24:30 5796 MB 2312822917 tb full E:\Data\Actual Projects
Gent.024 gntinfra1 08/10/06 09:24:30 75 GB 2312822917 hb full E:\Data\Actual Projects
Gent.018 gntinfra1 08/10/06 11:33:15 0 KB 2279276034 ci full E:\Data\Actual Projects

I don't see the indexes of the incremetal backup ?

root@edgbckprd01 / # mminfo -s edgbckprd01 -q "client=gntinfra1,name=C:\\" -o -vt
Gent.021 gntinfra1 08/06/06 20:05:14 1729 MB 1322659763 cb full C:\
Gent.019 gntinfra1 08/07/06 22:04:24 1730 MB 4275543330 cE full C:\
Gent.023 gntinfra1 08/08/06 00:17:44 2417 KB 936885347 cb incr C:\
Gent.023 gntinfra1 08/08/06 15:02:57 41 MB 4208495580 cb incr C:\
Gent.023 gntinfra1 08/08/06 23:04:08 27 MB 3755539619 cb incr C:\
Gent.003 gntinfra1 08/09/06 23:04:36 45 MB 752504384 cb incr C:\

May I conclude there is only a problem with full backups ?

ble1 · Answer

About the indexes, I thought after a incrementalbackup networker also saved a index. But with mminfoI don't see this entry. What mminfo query did you use?

nsrhti · Answer

I am still testing, testing and testing ...

When I change the saveset E:\Data\Actual Projects into another directory for example E:\Data\Closed_Projects everything runs without a problem.

new conclusion:

Corrupted files ?
Amount data too large ?

I opened a case at the local support, but they are still searching ...

About the indexes, I thought after a incremental backup networker also saved a index. But with mminfo I don't see this entry.

ble1 · Answer

According to mminfo only 2279276034 is not valid as it is incomplete (perhaps in progress). Not sure what you mean under 'I don't see indexes', but index saveset does not belong to client, but rather to server so you will need to change mminfo query to list those. I think this should be easy one for support if they get webex to your env...

nsrhti · Answer

Oops sorry I have mixed some commands
I f I do mminfo -q "name=index:gntinfra1" -ot ¿v I see all the indexes.

But If I do root@edgbckprd01 /nsr/index # mminfo -s edgbckprd01 -q "client=gntinfra1,name=E:\Data\Actual Projects" -ot -v

I don't see the incremental backup. See earlier post.

nsrhti · Answer

root@edgbckprd01 / # mminfo -s edgbckprd01 -q "client=gntinfra1,name=E:\Data\Actual Projects" -ot -v
volume client date time size ssid fl lvl name
Gent.023 gntinfra1 08/08/06 15:58:32 97 GB 4107835661 cb full E:\Data\Actual Projects
Gent.023 gntinfra1 08/08/06 18:07:17 97 GB 4091066291 cb full E:\Data\Actual Projects
Gent.018 gntinfra1 08/10/06 09:24:30 5796 MB 2312822917 tb full E:\Data\Actual Projects
Gent.024 gntinfra1 08/10/06 09:24:30 75 GB 2312822917 hb full E:\Data\Actual Projects
Gent.018 gntinfra1 08/10/06 11:33:15 0 KB 2279276034 ci full E:\Data\Actual Projects

I don't see the indexes of the incremetal backup.
If I do the same for another saveset I also see the incrementals, see below

root@edgbckprd01 / # mminfo -s edgbckprd01 -q "client=gntinfra1,name=C:\\" -o -vt
Gent.021 gntinfra1 08/06/06 20:05:14 1729 MB 1322659763 cb full C:\
Gent.019 gntinfra1 08/07/06 22:04:24 1730 MB 4275543330 cE full C:\
Gent.023 gntinfra1 08/08/06 00:17:44 2417 KB 936885347 cb incr C:\
Gent.023 gntinfra1 08/08/06 15:02:57 41 MB 4208495580 cb incr C:\
Gent.023 gntinfra1 08/08/06 23:04:08 27 MB 3755539619 cb incr C:\
Gent.003 gntinfra1 08/09/06 23:04:36 45 MB 752504384 cb incr C:\

ble1 · Answer

don't see the incremental backup. See earlier post.

When was your last incremental backup? Do following test, run incremental backup now. Then do mminfo -avot -t today -r ssid,name,client,level,ssflags,sumflags,volume (make sure that time on client is in sync with time on server). Before you run mminfo query, after backup, do nsrck -m and check daemon.log on server for any suspicious errors/warnings and that backup was successful.

ble1 · Answer

As I said before, index belongs to server so you need to change query (instead of gntinfra1 use edgbckprd01 as client and pipe findstr command for index and gntinfra1).

blochner1 · Answer

Based on the fact E:\Data\Closed_Projects runs OK but E:\Data\Actual Projects does not, I would try putting quotes around the save set name because of the space:

"E:\Data\Actual Projects" and if still doesn't work try "E:\\Data\\Actual Projects"

NetWorker

Full backup is performing more then once

Was this post helpful?