oracle archive log file corruption

Question

Hello All ,

I am Working as Solaris Unix System Administrator.

Recently we got a incident from oracle DBA stating that they are seeing archive log corruption frequently once in a day, once in a two days.

The oracle writing the archvie logs in two location (Veritas file systems) simantaneously on Solaris on SPARC and X86, they are seeing archvie log corruption one of the two file system.

We have compared both the file systems, no differences found.

We have analyzed the system logs, there is no evident for file system corruption and also ran file system check to make sure if there is any file system block corruption and to file system structure, but we did not find any issues.

The disks are coming from EMC Clarrion Array. We suspect may be a issue with EMC SAN. Please advice on this.

Error messaes from Oracle:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on ORA_DISK_1 channel at 05/28/2010 01:04:34
ORA-19587: error occurred reading 512 bytes at block number 1
ORA-27091: unable to queue I/O
ORA-27072: File I/O error

Regards,

Raj.

ble1 · Answer

Check Oracle's alert log.

rrk2 · Answer

Hi ,

there is a oracle archive log corruption recently in other server. Please find the requested logs.

corrupted file

==========

root:servername# ls -ltr ./oatldc1/oatldc1_1_9939_669894695.arc
-rw-r----- 1 oracle dba 512 Jun 22 01:50 ./oatldc1/oatldc1_1_9939_669894695.arc

checking file type

=============

root:servername# file ./oatldc1/oatldc1_1_9939_669894695.arc
./oatldc1/oatldc1_1_9939_669894695.arc: data

checking content of file

=================

Archive file log

===========

Tue Jun 22 01:50:13 2010
Thread 1 cannot allocate new log, sequence 9939
Private strand flush not complete
Current log# 2 seq# 9938 mem# 0: /u13/oradata/oatldc1/oatldc1_t1_g2_m1.rdo
Current log# 2 seq# 9938 mem# 1: /u14/oradata/oatldc1/oatldc1_t1_g2_m2.rdo
Tue Jun 22 01:50:15 2010
Thread 1 advanced to log sequence 9939 (LGWR switch)
Current log# 4 seq# 9939 mem# 0: /u13/oradata/oatldc1/oatldc1_t1_g4_m1.rdo
Current log# 4 seq# 9939 mem# 1: /u14/oradata/oatldc1/oatldc1_t1_g4_m2.rdo
Tue Jun 22 01:50:16 2010
ALTER SYSTEM ARCHIVE LOG
Tue Jun 22 01:50:16 2010
Thread 1 cannot allocate new log, sequence 9940
Private strand flush not complete
Current log# 4 seq# 9939 mem# 0: /u13/oradata/oatldc1/oatldc1_t1_g4_m1.rdo
Current log# 4 seq# 9939 mem# 1: /u14/oradata/oatldc1/oatldc1_t1_g4_m2.rdo

Regards,

Rajesh.

rrk2 · Answer

Hi , Last archive log file corruption on 15th of june, we don't have log for that.   I will let you know logs if there will any further archive log corruption. Regards, Rajesh.

ble1 · Answer

Rajesh, could you post alert log which is according to timestamp from RMAN when trying to delete archive log?

rrk2 · Answer

Hi , As requested, please find the attached archive log corrupt file. Regards, Rajesh.

ble1 · Answer

I don't see anything attached.&#xa0; Perhaps your uploaded log got corrupted too

coganb · Answer

Hi Rajesh, If you're having trouble attaching the necessary file, you can upload it to ftp.emc.com/outgoing/rrk and update here when you've done so. -Bobby

rrk2 · Answer

Hi ,

I have transferred the oracle corrupt log file to ftp.emc.com under /outgoing/rrk and the file name is oatldc1_oatldc1_1_9939_669894695.arc

ftp> mput *.arc
200 PORT command successful
150 Connecting to port 1623

226-102561.4 Mbytes free disk space
226-File successfully transferred
226 0.675 seconds (measured here), 0.74 Kbytes per second
ftp: 512 bytes sent in 0.00Seconds 512000.00Kbytes/sec.
ftp> dir *.arc
200 PORT command successful
150 Connecting to port 1644
-rw-r--r-- 1 50 1100 512 Jun 24 13:14 oatldc1_oatldc1_1_9939_669894695.arc

226-Options: -l
226 1 matches total
ftp: 94 bytes received in 0.00Seconds 94000.00Kbytes/sec.
ftp>

Regards,

Raj.

rrk2 · Answer

Hi ,

I have uploaded the file to ftp.emc.com under /outgoing/rrk and the file name is oatldc1_1_7049_669894695.arc

ftp> put oatldc1_1_7049_669894695.arc
200 PORT command successful
150 Connecting to port 59086
#
226-83883.3 Mbytes free disk space
226-File successfully transferred
226 0.110 seconds (measured here), 4.55 Kbytes per second
local: oatldc1_1_7049_669894695.arc remote: oatldc1_1_7049_669894695.arc
512 bytes sent in 0.11 seconds (4.75 Kbytes/s)

Regards,

Raj.

ble1 · Answer

Rather then your archive log we need alert log which matches timestamp from your RMAN error mentioned before.

rrk2 · Answer

Hi , Could you please update on this. Regards, Raj.

ble1 · Answer

What you uploaded is not Oracle's alert log!  Ask Oracle DBA to check RMAN log when this corruption happened last and along with that log suppy sbtio.log and Oracle's alert log.

NetWorker

oracle archive log file corruption

Was this post helpful?