Unsolved
This post is more than 5 years old
4 Posts
0
3351
oracle rman backup slow after write cache disabled/enabled on Clariion
Hello,
Couple weeks ago we had a vault disk failure on Clariion cx-380. As a result of it, write cache was disabled automatically. The big noticable impact was on oracle side. The rman database backup slowed down considerably. After the disk was replaced and rebuild was completed, write cache was enabled automatically. However, database backup is still very slow. What could be the reason for this ? It used to take 12hours for backup, now it takes 36 hours.
Host :
SUN E6900
Solaris 5.9
Oracle 10.0.2
Storage :
Clariion CX-380
Currently write cache and read cache are enabled.
SPs have been rebooted, host has been rebooted, oracle has been restarted, asm as been restarted, currently no tresspassed luns, no errors on storage, no errors on host, all power paths are active.
Thanks
LouisLu
161 Posts
0
November 21st, 2011 17:00
Hello,
The backup policy is sometimes tricky. We assumed the backup went to the disk assigned to Oracle first and then move to tape. Is it right? How much the data is backuped up?
To locate the issue further, we need to know the speed of normal DB queries, speed of other application runnining on the CX3 and SUN servers?
Luis
reseach
225 Posts
0
November 22nd, 2011 00:00
Could you let us know what designed backup architecture is in your environment?
The following is general consideration for your reference,
You might need to check your “large pool size” setting if you backup to host local/ attached storage.
You might need to check status of network health and backup device to find out where the bottle neck is, because RAMAN transfer data through Oracle Net, I prefer you check network first.
Thanks,
Eddy
cricket1201
4 Posts
0
November 22nd, 2011 11:00
-Backup is to the disk(SATA) only. It is a separate volume on the host file system but using the same storage. database is on FC disks.
- it is a datawarehouse system so the data being backuped is a lot maybe couple hundred gigs. we also have another system with same configuration and storage and there the backup runs in 12 hours. It used to take similar time on this problematic server as well but now it takes 36 hours.
- no tapes are used
- system is performing as before
- normal DB queries, reports etc are running as before, maybe a 'touch' slower.
- Load on Sun server has been normal apart from increased i/o when backup is running.
What I have noticed is that it is only the backup part that is slow. The recover part is taking almost the same time as before. Earlier backup part used to take 3 hours and recover part 9hours. Now the backup part is taking 26-27hours. This behaviour has only started since the write cache problem on the storage
Eddy, your questions are relevant only if configuration was changed on the server or the network. There has been no change apart from problems we had due to write cache on this system.
reseach
225 Posts
0
November 22nd, 2011 20:00
SP write cache re-enabled after vault drives fail is normal operation on CX, I did not see any performance impact of it upon my experience and searching ETA/KB.
Is your backup volume shared Raid group with any other application? Is your backup volume using vault drivers? Those would impact performance.
You also could contact your service people to collect NAR file to figure out what is your CX doing, if anything wrong, you could submit a SR to EMC GTS.
If all of SP, Cache,etc are doing well, @ least you could wipe off a suspect and look back to other components on backup IO path.
Eddy
LouisLu
161 Posts
0
November 23rd, 2011 00:00
Some assumption built on your symptoms---only backup part that is slow. Can you enable Enable Read and Write Cache at the LUN level for specific backup part? It is not enable write cache only.
Lu
houndfish
53 Posts
0
December 1st, 2011 06:00
Hi Cricket120,
you should really open an SR with Customer Service and have them check out the array. They will be happy to help.
Dave