Unsolved
This post is more than 5 years old
9 Posts
0
19658
October 23rd, 2004 06:00
Problem Fixed: Slow backup with PV132T and BackupExec 9.1
Hi all,
Just wanted to share a solution to a problem we experienced with slow backups using PowerVault 132T and BackupExec 9.1. I did quite a bit of searching on the forums and noticed that many people out there have had similar performance issues with using BE with various LTO-based tape libraries.
After connecting a shiny-new PV132T (dual LTO2) to shiny-new PE2650 server running Windows Server 2003 Standard and loading BackupExec 9.1 with all of the latest hotfixes and drivers we noticed that the performance for our backups averaged at 200mb/s, which is extremely slow for an LTO2. The verify part of the backup job, however, proceeded at a healthy 2000+ mb/s. The backups were performed in a test environment without any other load on the system and with only a single backup job to a single tape drive. In addition, the results were the same regardless of whether we were backing up a local or remote system and regardless of whether we used the latest tape drive drivers offered by Dell or Veritas. Changing the buffer and block size settings had no noticeable impact either.
After numerous calls to Dell and Veritas support it turned out that our local McAfee real-time anti-virus service was responsible for slowing down the backup. With Virusscan disabled our backup rates have jumped to over 1000 mb/s. We were advised by McAfee to exclude the BackupExec folder from being scanned, configure the service to start as a low-priority process, or to use the scheduler to shutdown the service prior to the backup window and restart it afterwards.
Subsequently changing the drive buffer setting to 256 has improved the performance by another 100 mb/s.
These are just some of our observations. Please check other factors other than the hardware and the backup software if you are experiencing a similar problem, a third-party background process might be to blame.
Hope this helps,
No Events found!
schneewe
15 Posts
0
November 12th, 2004 20:00
ed6612
9 Posts
0
November 13th, 2004 04:00
Hi Jonathan,
I was advised to exclude the backupexec folder from being actively scanned on the media server (our backup performance was slow even during media server backup) but that did not appear to solve the problem. We resolved this issue by configuring our mcafee on-access scanner to exclude backupexec-related executables from being scanned (bkupexec.exe, bengine.exe, beserver.exe, etc) and now our rates for running 2 concurrent jobs on 2 LTOII drives are around 1000mb/m each. This is a prefered method to completely shutting down the virus scanner during the backup window.
Glad to hear that Symantec's scanner is not affecting the backup rate although I wonder if having a SAN-based backup library vs a SCSI-based one has an effect. We have not so far really experimented with backup-to-disk on a large scale but it is surprizing that it would lag behind a linear tape drive. Other limiting factors must apply I guess (network bandwidth, source disk read speed, backup software itself)
Thanks for the post,
-Ed
schneewe
15 Posts
0
November 15th, 2004 10:00
Thanks for the detail Ed,
To be fair, I have not completed every possible BU2D scenario, but it general I've found the EMC ATA drives to be terribly slow. I've tried everything I can thing of -- file system block alignment with the RAID5 stripe element, changing the NTFS allocation unit (typically use 64K on Win2K3), changing the RAID5 Element size (usually 128 sectors = 64K). In general I'm getting about <5MB/s per drive! Typically 20-30MB/s sustained for 7 Disk (6+1) RAID5. -- This is with one LUN only per RAID group so no contention.
The one thing I have not tried is changing the number of RAID5 columns (5 Disk vs. 9 Disk). Since EMC's recommended ATA SOP is one HotSpare per DAE, I've been using 7 Disk (6+1) RAID5.
I've been extremely impressed with the performance and reliability of my new PV136T versus my aging ADIC Scalar 24 SDLT320 and BU2D. Granted, my is library is SAN attached, but as I've posted in some other threads, 90% of the time I'm source performance limited rather than tape write performance limited.
One thing I tried over the weekend was backing up using EMC SnapView. If you're not familiar this Clariion based software it takes a "snapshot" of a production LUN/LUNs (in my case Oracle), mounts them to the media server, and then backup proceeds via BEX as if local disk (no IP traffic).
CX600 Prod LUN -> SnapShot -> SAN attached MediaServer -> SAN attached Tape
Here's a comparison of IP vs. SnapView
(Both are to LTO2 on my PV136T w/BE9.1 SP1 w/Dell device drivers)
IP Based Backup to Tape (dual gigabit on host and media server):
Elapsed time : 05:44:01
Files : 213
Directories : 56
Byte count : 449,191,443,479 bytes
Job rate : 1,877.00 MB/Min (Byte count divided by Elapsed time for job)
AKA 30MB/s
SnapView SAN based backup of the same data:
Elapsed time : 03:54:07
Files : 213
Directories : 56
Byte count : 449,191,419,143 bytes
Job rate : 3,616.00 MB/Min (Byte count divided by Elapsed time for job)
AKA 60MB/s
This dataset was EXTREMELY compressible (~30:1) as Oracle tablespaces are typically pretty sparse. Unfortunately, using SnapView requires so many carefully coordinated scripts so I've not implemented this in production yet.
I'm planning on repeating this test using SnapView only comparing backup to LTO2 vs. BU2D. Then we'll see who's the champ!
I was hoping to leverage Jumbo frames to speed my IP backups and other large file transfers (our users move a lot of data around from subnet to subnet) but I haven't been able to get this to work in my environment. In general I've found our server's gigabit performance to be rather lacking -- for a single session seems to max out around 250-400Mbit. I know its not our switch gear (Nortel 5510) these babies have a 72Gbit fabric and we have all new wiring. Seems like a Win2K3, TCP/IP stack, or Broadcom teaming issue.
-Jonathan
ed6612
9 Posts
0
November 19th, 2004 04:00