since upgrading Networker from 7.6.5 to 8.1.1.4, the simple file-level LAN backup of the large Windows file server intermittently fails when tape full and a next spanning, continuation tape is started

Hi Geza,

I do realize you did many tests and that everything points to source you mentioned, but still let me ask few things:

a) did you started to see this when client was updated or both client and server (and storage nodes)?  If client, then it may be possible to test few other versions of clients to see if same issues appears

b) quit signal is something I usually see with slower clients (WAN) running AV.  Changing policy to avoid read scans or excluding save from scan helps (and in specific case of Symantec we noticed that its own self-protection would kill save when going over its AV signature files).  I'm not stating this is your problem, but different NW versions may have different behavior pattern which may or may not be suspicious to AV so if your server is running AV it might be good thing to check it out.  As in your case, I have seen this happening on more or less random occasions and never really understood why.

c) keepalive you already mentioned, but usually this is required on both sides of communication channel and not just client - I doubt this is your issue, but worth exploring.

d) do you see the same thing if you kick the backup from the client side (I realize that this will be pain to test due to size and point that this does not happen always, but usually this test - if confirmed to be successful -  indicates that control session was killed. In your case I suspect it is data session which gets killed and not control one, but it would be nice to complete picture to have this tested too.

e) do you use TCP offloading on SW level?

Hi,

thanks for your quick response.

Q a) did you started to see this when client was updated or both client and server (and storage nodes)?  If client, then it may be possible to test few other versions of clients to see if same issues appears

A: We believe that it happened when client updated. Server was updated end of March or so, client around beginning of May. We are also thinking about either downgrading client (back to "original" 7.6.5 that worked properly with NW Server 7.6.5 for quite some time, and it even worked one month quite well with NW Server 8.1.1), or update the client to the latest fixes (8.1.1.6 is already available). Actually, based on the compatibility info we have from docs (Install Guide?), newer server should be OK bwith earlier client, at least with these specific versions (8.1.1 and 7.6.5).

Q b) quit signal is something I usually see with slower clients (WAN) running AV.  Changing policy to avoid read scans or excluding save from scan helps (and in specific case of Symantec we noticed that its own self-protection would kill save when going over its AV signature files).  I'm not stating this is your problem, but different NW versions may have different behavior pattern which may or may not be suspicious to AV so if your server is running AV it might be good thing to check it out.  As in your case, I have seen this happening on more or less random occasions and never really understood why.

A: Well, yes, as this is a typical Windows file server, it has server-side AV installed. (McAfee McShield, v 8.8 ?) Your idea (regarding AV and its settings) could help, an EMC support engineer also recommended that (i.e. some AV exclude rules/directories, no scan on backup file opens/reads). We will go after this hint - unfortunately it takes some extra time as the security officer who has permissions to change AV settings is now on holiday.

The connectivity is LAN, relatively low latency 1G, so communications should not be an issue.

Q c) keepalive you already mentioned, but usually this is required on both sides of communication channel and not just client - I doubt this is your issue, but worth exploring.

A: That may make sense, network parameter tuning (e.g. keepalive) was "only" done on the client side.

Q d) do you see the same thing if you kick the backup from the client side (I realize that this will be pain to test due to size and point that this does not happen always, but usually this test - if confirmed to be successful -  indicates that control session was killed. In your case I suspect it is data session which gets killed and not control one, but it would be nice to complete picture to have this tested too.

A: A-ha. I think I understand what you mean, but let us confirm. So you mean a client-initiated backup (e.g. save command, Networker User GUI / Backup) does the data stream only, and the control stream (that is also there when doing a server-initiated backup, so running a group) in that case is not part of the equation... Did I understand your idea well? Well, we have not tried client-initiated backups so far to test this issue. Even with server-initiated backups, and we ran an awful lot of test save groups, it is pretty uncertain whether we see the error or not, and complicated and time-consuming to test. Our impression is, that by running only one saveset at a time, e.g. as a test, we usually do not see the problem, even when >1.5TB and some 2m files backed up, there are 3-4-5 volumes involved and volume full, volume spanning happens during the backup. When running multiple of those savesets at the same time, at level=full (this is how "normal" weekend backups should look like), we almost always get the problem, some 16-18 volumes consumed, >10 volume full events occuring, and out of those occurences, at least one such event generates the error. We experienced even this: multiple savestreams going to the same physical tape drive (multiplexed), tape full, a next tape loaded (and probably (re-)labelled), and all but one savestreams continue to back up data, with the remaining one savestream failing at this point in time.

Q e) do you use TCP offloading on SW level?

A: no offloading, not to my knowledge.

Regards,

Geza

• Yes, your client version should be ok though I would rather use NW8.x with NW8.x server.  You can also try 8.0.3.x for example.
• Yes, you are correct - client initiated backup won't have idle control session while data stream is active.  In past (and present I guess) that used to be an issue and there were various workarounds like enabling verbose on group or setting NSR_KEEPALIVE_WAIT in startup scripts (now /nsr/nsrrc in NW8, but on Windows still in ENV variables). From what you said so far, it doesn't look like an issue with control session, but rather data session.
• I'm wondering how much a fact that this happens after tape change is a clue or misleading. Could be both I guess, but it is more likely a pattern if it happens in all your tests/runs.  Based on your logs - does this happen after:
  • tape filled, next tape requested, next tape relabeled, tape started to be used, kaboom, or
  • tape filled, next tape requested, next tape loaded, tape started to be used, kaboom,
  • does not matter

In third point, different is that next tape is either relabeled or already relabeled/new tape is loaded. This, normally, should not matter at all, but it seems like we deal with rather specific issue so specific condition may apply too.  If, and that's big if, if it would turn out that this happens only after tape is being relabeled first, then next question would be how much were you waiting for tape to be changed until job could continue (as waiting time might be crucial here at TCP level when dealing with idle sessions).

Hi,

• Q:...I would rather use NW8.x with NW8.x server.  You can also try 8.0.3.x for example.
  • A: We were thinking about "downgrading" to 7.6.5 as that was the version in use (both on server and client) before going to 8.1.1
• Q: does this happen after:
  • tape filled, next tape requested, next tape relabeled, tape started to be used, kaboom, or
  • tape filled, next tape requested, next tape loaded, tape started to be used, kaboom,
  • does not matter

In third point, different is that next tape is either relabeled or already relabeled/new tape is loaded. This, normally, should not matter at all, but it seems like we deal with rather specific issue so specific condition may apply too.  If, and that's big if, if it would turn out that this happens only after tape is being relabeled first, then next question would be how much were you waiting for tape to be changed until job could continue (as waiting time might be crucial here at TCP level when dealing with idle sessions).

A: this mostly happens with load only, no need for re-label. (This is because the backup admin has sufficient number of empty, labelled virtual tape volumes readily available.) But we have also seen the same error a few times when Networker automatically re-labelled an expired volume. Timing is not necessarily (I would say, probably not) an issue: in case of the VTL, there are 16 virtual tape drives and one or two always immediately available to continue backup, and virtual tape volumes also available right away. Furthermore, loading and even re-labelling a VTL is much faster than with physical tape - we usually see less than 1 minute between volume full and backup continued on the next vol, even with labelling included. And in most of the cases, in less than a minute from vol full (so only a few sec after writing continues) the savestream already fails...

ok, it is probably fair to say that timing between stop and start of ongoing session should not be an issue then (well, one way to test it would be to have continuous tests to pure disk device, but I understand this is not your setup so this is not possible to test).