UNSOLVED
A
antoniogray
6 Posts
0
423
May 3rd, 2013 08:00
SQL plug-in and long SQL I/O requests
Running into an issue with the SQL module when running full backups. I'm working with EMC support on this issue, but figured I'd see if anyone else might have come across something similar.
Our issue, once a full DB backup starts, the communication between Avamar -> SMO -> DB seemingly makes so many requests to the DB it is attempting to back up, it generates several I/O service time messages from SQL for that specific DB. In turn the databases become extremely sluggish or unresponsive to any other user or job trying to access the DB, but the backup will always complete successfully.
We are using the SQL plug-in module to backup up a couple of servers, each having 2 - 5 databases on them, total data volume might be around 200GB - 400GB on each server. SQL starts complaining 'SQL Server has encountered occurrence(s) of I/O requests taking about 15 minutes after the backup starts and then we'll see these messages consistently throughout the duration of the backup.
We are working through the process of checking server resources, seemingly there is far, far more CPU and physical memory than is needed, and we are in the process of getting some performance info from the SAN team (fibre to a VMAX). The DBA and app teams can both generate a significant amount of load on each of these servers/databases for extended periods of time, hours, and not one warning message.
Running 6.1.1-87 on the grid and client and confirmed our environment has met all the requirements as listed in the Avamar guides. This issue began when we upgraded to the 6.1 release; I’m constantly being asked if I can migrate back to the 6.0 version and/or entirely back to Netbackup.
Thanks,
-Antonio
antoniogray
6 Posts
159
0
Posted May 7th, 2013 12:00
Thanks a bunch for the reply. I've tried various stream and buffer sizes and all yield the same result. Interesting thought about the file system unit size, will certainly check that out. And hopeful there isn't some underlying issue but who knows.... looking into that now. But I have a couple of environments which I can test with but it's hard for me to wrap my head around the fact that performing a backup of one DB at a time, even on a system with very little load, my 'test environment', would put us in this situation. I'll be testing a bit tonight and will watching permon closely. Also going to try an older 6.1 version of the client as see if that turns up anything different.
fdxpilot
2 Intern
•
143 Posts
159
0
Posted May 3rd, 2013 12:00
Are you using more than one stream in Avamar when you backup these SQL servers? If so, you might try reducing the backup stream count and see if that results in an acceptable level of host IO. It would also be interesting to watch the physical disk performance counters in perfmon while a backup is running; watch for split IOs and response times. Also, double check SQL storage best practices and see if anything is configured suboptimally. I've seen big jumps in performance from formatting NTFS volumes with a 64 KB allocation unit size instead of 4 KB, for instance. Is each database broken out into multiple separate storage volumes (data, tempdb, logs, etc)? Are multiple databases which are being backed up at the same time on the same back-end disks? Backups can be very IO intensive and expose other issues that have been lurking. Does a native SQL backup to a fast local disk generate a comparable amount of IO on the source database volumes?