Unsolved
This post is more than 5 years old
2 Intern
•
1.1K Posts
0
992
April 3rd, 2009 05:00
Backup of one disk is successful after increasing parallelism
This is an interesting one and I am wondering if anyone has any ideas why my resolution worked. It is logged to EMC's Sesame Street office who have yet to understand the problem enough to give an intelligible response.
We have a Windows box with a Sharepoint database on the D: drive along with a big pile of other data on the same drive, C: drive of a typical size, VSS system savesets. Parallelism set to 2. Server parallellism is 128 - testing was done during the day when server was quiet. C: and all VSS but for VSS SYSTEM SERVICES: are always successful, VSS SYSTEM SERVICES: and D: would start to backup but would always fail. Initially the problem was blamed on VSS or Sharepoint or a mix of the two. Backup is to an EDL emulating LTO1 drives.
I did the following reconfiguration: used savepnpc to take down the Sharepoint database and back it up; since the bulk of D: data is in D:\Common I used a directive to skip that data giving us 2 hours to backup about 20GB Sharepoint and 20GB VSS data, and configured a new backup of the D:\Common directory to run earlier in the night (about 80GB starting 7 hours before the savepnpc job).
I didn't expect this and it didn't... However I found the version of the client was only 7.2.0 so we upgraded to 7.4.4 After this all VSS savesets were successful, both backups of the D: drive still failed. Doing some further testing backup ran okay for a while then said it was finished and started again - no errors or timeouts even with savegrp -D9 output.
I also tried disabling VSS backups with the same result. Looking at mminfo output I noticed that for maybe 80-90% of the backups the time difference between the start and end of the backup was 2 hours to the minute which seemed more than coincidental (this is excluding scheduled savepnpc backups which have a timeout after 2 hours). Several of the other savesets were 90 minutes to the minute (there is no sequential results here which would suggest some parameter had been amended) and the other savesets may have been manually stopped. However I had no timeouts corresponding to these values and the error messages never indicated a timeout.
As I am currently aware the D: is just a regular disk drive, but I thought I would push the parallelism up and see whether that had any effect; my thought was if I can get data off a little faster then maybe I could get the shorter savepnpc to backup in the time window. However, I did run a manual backup on D:\Common to see if we saw any improvement in performance, which we did not (throughput was about 10Mb/s still) but when I came back to it after 140 minutes it was still running (about 50GB backed up, in 2 hours we usually saw about 30-50GB backed up) - after 3.5 hours it finished backing up 80GB and that night the savepnpc/Sharepoint data backed up.
So I want to know how this could fix the problem. The way it looks we have when D:\Common backs up, one stream of data so the parallelism setting should not be restrictive even though it is a low value of 2 and I would not expect any effect on increasing this value to 12 (or any other higher value). Even if D: had multiple disks and could support a higher parallelism value I cannot see that this would prevent us being able to backup if the parallelism was lower. All in all I would not expect changing the parallelism would lead to success of the backup except if more streams could run, therefore backups quicker, therefore backups take place in less time, if this fits into 2 hours we get a successful backup.
The implication is that a bit more may be going on with parallelism than the official line states... Anyone have any ideas on this one?
We have a Windows box with a Sharepoint database on the D: drive along with a big pile of other data on the same drive, C: drive of a typical size, VSS system savesets. Parallelism set to 2. Server parallellism is 128 - testing was done during the day when server was quiet. C: and all VSS but for VSS SYSTEM SERVICES: are always successful, VSS SYSTEM SERVICES: and D: would start to backup but would always fail. Initially the problem was blamed on VSS or Sharepoint or a mix of the two. Backup is to an EDL emulating LTO1 drives.
I did the following reconfiguration: used savepnpc to take down the Sharepoint database and back it up; since the bulk of D: data is in D:\Common I used a directive to skip that data giving us 2 hours to backup about 20GB Sharepoint and 20GB VSS data, and configured a new backup of the D:\Common directory to run earlier in the night (about 80GB starting 7 hours before the savepnpc job).
I didn't expect this and it didn't... However I found the version of the client was only 7.2.0 so we upgraded to 7.4.4 After this all VSS savesets were successful, both backups of the D: drive still failed. Doing some further testing backup ran okay for a while then said it was finished and started again - no errors or timeouts even with savegrp -D9 output.
I also tried disabling VSS backups with the same result. Looking at mminfo output I noticed that for maybe 80-90% of the backups the time difference between the start and end of the backup was 2 hours to the minute which seemed more than coincidental (this is excluding scheduled savepnpc backups which have a timeout after 2 hours). Several of the other savesets were 90 minutes to the minute (there is no sequential results here which would suggest some parameter had been amended) and the other savesets may have been manually stopped. However I had no timeouts corresponding to these values and the error messages never indicated a timeout.
As I am currently aware the D: is just a regular disk drive, but I thought I would push the parallelism up and see whether that had any effect; my thought was if I can get data off a little faster then maybe I could get the shorter savepnpc to backup in the time window. However, I did run a manual backup on D:\Common to see if we saw any improvement in performance, which we did not (throughput was about 10Mb/s still) but when I came back to it after 140 minutes it was still running (about 50GB backed up, in 2 hours we usually saw about 30-50GB backed up) - after 3.5 hours it finished backing up 80GB and that night the savepnpc/Sharepoint data backed up.
So I want to know how this could fix the problem. The way it looks we have when D:\Common backs up, one stream of data so the parallelism setting should not be restrictive even though it is a low value of 2 and I would not expect any effect on increasing this value to 12 (or any other higher value). Even if D: had multiple disks and could support a higher parallelism value I cannot see that this would prevent us being able to backup if the parallelism was lower. All in all I would not expect changing the parallelism would lead to success of the backup except if more streams could run, therefore backups quicker, therefore backups take place in less time, if this fits into 2 hours we get a successful backup.
The implication is that a bit more may be going on with parallelism than the official line states... Anyone have any ideas on this one?
No Events found!


ble1
6 Operator
•
14.4K Posts
•
56.2K Points
0
April 3rd, 2009 05:00
DavidHampson
2 Intern
•
1.1K Posts
0
April 3rd, 2009 08:00
There is not an error when D: fails, it said D: finished then restarted backing up the data.
David
DavidHampson
2 Intern
•
1.1K Posts
0
April 3rd, 2009 08:00
D: drive is 153GB used 56GB free, mirrored disks containing the C: and D: partitions.
Hari5
443 Posts
0
April 12th, 2009 07:00
DavidHampson
2 Intern
•
1.1K Posts
0
April 15th, 2009 02:00
ble1
6 Operator
•
14.4K Posts
•
56.2K Points
0
April 15th, 2009 11:00
Just out of curiosity, did you try ftp test for that slow session which fails? So, in your case, you would need to run ftp from storage node to client and do mget on parent folder. Do you get same thing then?