
Attachment uploads are currently disabled. This is a temporary situation and will resume as normal in the coming days.
UNSOLVED
Issues with MD3000i and Backup Exec Oracle Agent
Hi all, we are setting up an Oracle RAC with ASM and testing out our backup/recovery process. We have two R900's as the DB servers, and an MD3000i + two MD1000 shelves for our SAN between them. The MD3000i is updated to firmware 07.35.22.60. Oracle is 10G Enterprise. Symantec is 12.5 with the Oracle Agent.
I am basically able to get a good backup of the database and able to do a full restore from a wiped database so it is TECHNICALLY working... Just at a snails pace. On the Oracle end of things, it is compressing about 44GB down to around 7.5GM, and compression on the actual Backup Exec job is disabled. Our MD3000i is a dual controller, so I have 4 CAT-6 cables split between two PowerConnect 5448's and the only other network connections to it are a management port and two more CAT-6 runs from two Broadcom 5709 NIC's in each of the two Oracle nodes.
I have created a test virtual disk on the same SAN and copied about 10GB of both large and small files up to this partition and it is mapped through to one of the two Oracle nodes as another hard drive. Our Backup Exec Media Server is a new Dell 2900 that is also connected up at Gb (currently teamed NIC's but about to try no teaming) to another Cisco switch where the "public" RAC interface (also teamed at the moment) is physically connected. If I create a Backup Exec job to pull down these 10GB or so of files off the SAN by way of the Oracle node, I get about 1400MB/min throughput when all is said and done. The MD3000i logs about 20 - 30 dropped frames for this 10GB transfer and most of the traffic stays on two of the 4 iSCSI connections.
If I configure an Oracle job that follows the exact same network path but (obviously) uses the Symantec Backup Exec Agent for Oracle, it is a different story. I realize that there is extra overhead from Oracle doing its own compression and reading from ASM, etc, but my overall throughput according to Backup Exec for about a 7.5GB total file transfer (again, compressd on the Oracle side from about 44GB) winds up being around 100MB/min. If I watch the MD3000i dropped frame counter while this job takes place, it completely SPEWS dropped frames... at the rate of a couple hundred ever second or two for a total of 80,000 - 100,000 dropped frames for this 7.5GB transfer. Unlike the "flat file" backup test above where most of the traffic stays lumped on two of the four iSCSI connections, the Oracle job very evenly distributes the load and the errors between all 4 iSCSI connections. Almost like it is causing the iSCSI connections to drop and recover, but I'm not seeing any actual evidence of that in Event Viewer (not sure exactly where else I might verify that).
Also tried copying (drag and drop through Windows explorer) large (3 - 4 GB) blocks of small files and a couple of large 7 - 8GB files across from the server where Backup Exec lives up through one of the Oracle nodes to the virtual SAN drive and they flew through without any dropped frames. Even tried copying along that same path at the same time we pulled down a large group of files in the opposite direction to create a bunch of concurrent read/writes on both ends, and it breezed through that as well. So I am relatively comfortable with at least the underlying networks that tie all of this together, but as of yet haven't ruled anything out.
Anyone experienced anythign similar? Going to be working with Symantec next, but wanted to parallel it here.
Thanks!
Responses (0)
Solutions (0)
