Unsolved

This post is more than 5 years old

2 Intern

 • 

1.1K Posts

992

April 3rd, 2009 05:00

Backup of one disk is successful after increasing parallelism

This is an interesting one and I am wondering if anyone has any ideas why my resolution worked. It is logged to EMC's Sesame Street office who have yet to understand the problem enough to give an intelligible response.

We have a Windows box with a Sharepoint database on the D: drive along with a big pile of other data on the same drive, C: drive of a typical size, VSS system savesets. Parallelism set to 2. Server parallellism is 128 - testing was done during the day when server was quiet. C: and all VSS but for VSS SYSTEM SERVICES: are always successful, VSS SYSTEM SERVICES: and D: would start to backup but would always fail. Initially the problem was blamed on VSS or Sharepoint or a mix of the two. Backup is to an EDL emulating LTO1 drives.

I did the following reconfiguration: used savepnpc to take down the Sharepoint database and back it up; since the bulk of D: data is in D:\Common I used a directive to skip that data giving us 2 hours to backup about 20GB Sharepoint and 20GB VSS data, and configured a new backup of the D:\Common directory to run earlier in the night (about 80GB starting 7 hours before the savepnpc job).

I didn't expect this and it didn't... However I found the version of the client was only 7.2.0 so we upgraded to 7.4.4 After this all VSS savesets were successful, both backups of the D: drive still failed. Doing some further testing backup ran okay for a while then said it was finished and started again - no errors or timeouts even with savegrp -D9 output.

I also tried disabling VSS backups with the same result. Looking at mminfo output I noticed that for maybe 80-90% of the backups the time difference between the start and end of the backup was 2 hours to the minute which seemed more than coincidental (this is excluding scheduled savepnpc backups which have a timeout after 2 hours). Several of the other savesets were 90 minutes to the minute (there is no sequential results here which would suggest some parameter had been amended) and the other savesets may have been manually stopped. However I had no timeouts corresponding to these values and the error messages never indicated a timeout.

As I am currently aware the D: is just a regular disk drive, but I thought I would push the parallelism up and see whether that had any effect; my thought was if I can get data off a little faster then maybe I could get the shorter savepnpc to backup in the time window. However, I did run a manual backup on D:\Common to see if we saw any improvement in performance, which we did not (throughput was about 10Mb/s still) but when I came back to it after 140 minutes it was still running (about 50GB backed up, in 2 hours we usually saw about 30-50GB backed up) - after 3.5 hours it finished backing up 80GB and that night the savepnpc/Sharepoint data backed up.

So I want to know how this could fix the problem. The way it looks we have when D:\Common backs up, one stream of data so the parallelism setting should not be restrictive even though it is a low value of 2 and I would not expect any effect on increasing this value to 12 (or any other higher value). Even if D: had multiple disks and could support a higher parallelism value I cannot see that this would prevent us being able to backup if the parallelism was lower. All in all I would not expect changing the parallelism would lead to success of the backup except if more streams could run, therefore backups quicker, therefore backups take place in less time, if this fits into 2 hours we get a successful backup.

The implication is that a bit more may be going on with parallelism than the official line states... Anyone have any ideas on this one?

6 Operator

 • 

14.4K Posts

 • 

56.2K Points

April 3rd, 2009 05:00

I didn't get what error do you get when D or SYSTEM SERVICES fails?

2 Intern

 • 

1.1K Posts

April 3rd, 2009 08:00

Hi Hrvoje

There is not an error when D: fails, it said D: finished then restarted backing up the data.

David

2 Intern

 • 

1.1K Posts

April 3rd, 2009 08:00

Further info - 100Mb card; still waiting to confirm duplex setting autodetected.

D: drive is 153GB used 56GB free, mirrored disks containing the C: and D: partitions.

443 Posts

April 12th, 2009 07:00

Don't have a clue ... But this sounds really interesting. Do let me know if you get a resolution on this one.

2 Intern

 • 

1.1K Posts

April 15th, 2009 02:00

A better response than I get from EMC's support! Reducing the parallelism to 1, 2 backup always fails. Backing up a smaller folder is always successful even on paralellism of 1. Some failure of full backup with a higher parallelism (4,12) but when failure occurs it is not after 2 hours.

6 Operator

 • 

14.4K Posts

 • 

56.2K Points

April 15th, 2009 11:00

Well, the only thing that comes to mind which might not be even close what you have is the fact that whenever I had slow backups (well, a stream) I would get an issue similar to yours. For some reason if stream could not be faster that let's say 1MB/s I would see something as you. Increasing parallelism would mask this if there would be more sessions and it would be fine sometimes. Usually, resolution was to fix the speed for faulty session. Same as you, client initiated backup would successful even sometimes it would be slow as well (not that slow, but still slow). So, I have seen symptoms, but the cause might be completely different. In all cases, as it seems, server initiated backup has problems to keep up with slow sessions or network connectivity is such that it breaks at certain point (could explain difference between server and client initiated).

Just out of curiosity, did you try ftp test for that slow session which fails? So, in your case, you would need to run ftp from storage node to client and do mget on parent folder. Do you get same thing then?
No Events found!

Top