Highlighted
2 Bronze

存储突然链接失败,请教如何查找原因?

各位好!

今天发现这样一个问题,我的同事反馈使用PL/SQL Developer登录oracle数据库失败,然后我登录服务器发现oracle实例未启动,于是尝试启动

SQL> startup

ORACLE instance started.

Total System Global Area 4764729344 bytes

Fixed Size 2027160 bytes

Variable Size 419430760 bytes

Database Buffers 4328521728 bytes

Redo Buffers 14749696 bytes

ORA-00205: error in identifying control file, check alert log for more info

SQL>

根据提示,我查看alert日志

Sun Oct 6 21:08:37 2013

Thread 1 advanced to log sequence 357916

Current log# 3 seq# 357916 mem# 0: /clusdsk/oracle/oradata/ORCL/redo03.log

Sun Oct 6 21:26:27 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_lgwr_17049.trc:

ORA-00345: redo log write error block 41385 count 2

ORA-00312: online log 3 thread 1: '/clusdsk/oracle/oradata/ORCL/redo03.log'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 41385

Additional information: -1

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_ckpt_17051.trc:

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control03.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control02.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control01.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_ckpt_17051.trc:

ORA-00221: error on write to control file

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control03.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control02.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control01.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

Sun Oct 6 21:26:28 2013

KCF: write/open error block=0x29497 online=1

file=19 /clusdsk/oracle/oradata/ORCL/TBS12.dbf

error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 169111

Additional information: -1'

Sun Oct 6 21:26:28 2013

KCF: write/open error block=0x294a7 online=1

file=19 /clusdsk/oracle/oradata/ORCL/TBS12.dbf

error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 169127

Additional information: -1'

Sun Oct 6 21:26:28 2013

CKPT: terminating instance due to error 221

Sun Oct 6 21:26:28 2013

Automatic datafile offline due to write error on

file 19: /clusdsk/oracle/oradata/ORCL/TBS12.dbf

Sun Oct 6 21:26:28 2013

Automatic datafile offline due to write error on

file 19: /clusdsk/oracle/oradata/ORCL/TBS12.dbf

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_dbw1_17047.trc:

ORA-00221: error on write to control file

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_dbw0_17045.trc:

ORA-00221: error on write to control file

Sun Oct 6 21:26:29 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_lgwr_17049.trc:

ORA-00340: IO error processing online log 3 of thread 1

ORA-00345: redo log write error block 41385 count 2

ORA-00312: online log 3 thread 1: '/clusdsk/oracle/oradata/ORCL/redo03.log'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 41385

Additional information: -1

Sun Oct 6 21:26:34 2013

Instance terminated by CKPT, pid = 17051

然后,我感觉是和存储链路有关

于是,访问磁盘试试看

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 29G 5.5G 22G 21% /

/dev/cciss/c0d0p5 89G 5.1G 80G 7% /home

/dev/cciss/c0d0p1 99M 20M 75M 21% /boot

tmpfs 3.8G 0 3.8G 0% /dev/shm

none 3.8G 104K 3.8G 1% /var/lib/xenstored

/dev/sda1 825G 164G 619G 21% /clusdsk

$ cd /clusdsk

$ ls

ls: reading directory .: Input/output error

$

接着,我又查看了系统日志

# cat messages

Oct 6 04:03:24 dbs-n2 syslogd 1.4.1: restart.

Oct 6 21:26:11 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP DOWN detected (2 6 0).

Oct 6 21:26:27 dbs-n2 kernel: rport-0:0-0: blocked FC remote port time out: saving binding

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 214919

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 396623951

Oct 6 21:26:27 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 49577986

Oct 6 21:26:27 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 398196815

Oct 6 21:26:27 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 49774594

Oct 6 21:26:27 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:27 dbs-n2 kernel: Aborting journal on device sda1.

Oct 6 21:26:27 dbs-n2 kernel: EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted

Oct 6 21:26:27 dbs-n2 last message repeated 2 times

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12423

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 1545

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 63

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 0

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: EXT3-fs error (device sda1) in ext3_dirty_inode: Journal has aborted

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 63

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 0

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: ext3_abort called.

Oct 6 21:26:28 dbs-n2 kernel: EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal

Oct 6 21:26:28 dbs-n2 kernel: Remounting filesystem read-only

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LIP reset occured (f8f7).

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LIP occured (f8f7).

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP UP detected (4 Gbps).

Oct 6 21:48:04 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP DOWN detected (2 6 0).

Oct 6 21:48:20 dbs-n2 kernel: rport-0:0-0: blocked FC remote port time out: saving binding

Oct 6 22:33:44 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:33:44 dbs-n2 kernel: end_request: I/O error, dev sda, sector 1050153047

Oct 6 22:33:44 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:33:44 dbs-n2 kernel: end_request: I/O error, dev sda, sector 1050153047

Oct 6 22:38:31 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:38:31 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12375

Oct 6 22:39:33 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:39:33 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12375

Oct 6 22:41:44 dbs-n2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found

Oct 6 22:41:47 dbs-n2 modclusterd: shutdown succeeded

Oct 6 22:41:48 dbs-n2 ricci: shutdown succeeded

Oct 6 22:41:49 dbs-n2 smartd[6631]: smartd received signal 15: Terminated

Oct 6 22:41:49 dbs-n2 smartd[6631]: smartd is exiting (exit status 0)

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Got SIGTERM, quitting.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth1.IPv6 with address fe80::6ab5:99ff:fe78:7f24.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth1.IPv4 with address 10.9.0.21.

Oct 6 22:41:49 dbs-n2 oddjobd: oddjobd shutdown succeeded

Oct 6 22:41:50 dbs-n2 libvirtd: 22:41:50.025: warning : Shutting down on signal 15

Oct 6 22:41:50 dbs-n2 heartbeat: : WARN: Shutdown delayed until current resource activity finishes.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Heartbeat shutdown in progress. (4753)

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Received shutdown notice from 'dbs-n1'.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Resource takeover cancelled - shutdown in progress.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Giving up all HA resources.

Oct 6 22:41:56 dbs-n2 ResourceManager[9308]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:41:56 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:42:00 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: Trying to unmount /clusdsk

Oct 6 22:42:01 dbs-n2 kernel: ext3_abort called.

Oct 6 22:42:01 dbs-n2 kernel: EXT3-fs error (device sda1): ext3_put_super: Couldnt clean up the journal

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: unmounted /clusdsk successfully

Oct 6 22:42:01 dbs-n2 Filesystem[9458]: INFO: Success

Oct 6 22:42:01 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:42:01 dbs-n2 IPaddr[9588]: INFO: ifconfig eth0:0 down

Oct 6 22:42:01 dbs-n2 IPaddr[9559]: INFO: Success

Oct 6 22:42:01 dbs-n2 heartbeat: : info: All HA resources relinquished.

Oct 6 22:42:02 dbs-n2 heartbeat: : info: Link dbs-n1:eth1 dead.

Oct 6 22:42:02 dbs-n2 ipfail: : info: Link Status update: Link dbs-n1/eth1 now has status dead

Oct 6 22:42:02 dbs-n2 heartbeat: : info: killing /usr/lib64/heartbeat/ipfail process group 4791 with signal 15

Oct 6 22:42:03 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:42:03 dbs-n2 ipfail: : info: Checking remote count of ping nodes.

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBFIFO process 4768 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBWRITE process 4769 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBREAD process 4770 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBWRITE process 4771 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBREAD process 4772 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4768 exited. 5 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4769 exited. 4 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4771 exited. 3 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4772 exited. 2 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4770 exited. 1 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: dbs-n2 Heartbeat shutdown complete.

Oct 6 22:42:04 dbs-n2 logd: : info: Waiting for pid=4660 to exit

Oct 6 22:42:04 dbs-n2 logd: : info: logd_term_write_action: received SIGTERM

Oct 6 22:42:04 dbs-n2 logd: : info: Exiting write process

Oct 6 22:42:05 dbs-n2 logd: : info: Pid 4660 exited

Oct 6 22:42:05 dbs-n2 saslauthd[6606]: server_exit : master exited: 6606

Oct 6 22:42:06 dbs-n2 xinetd[4645]: Exiting...

Oct 6 22:42:10 dbs-n2 hcid[4374]: Got disconnected from the system message bus

Oct 6 22:42:11 dbs-n2 rpc.statd[4295]: Caught signal 15, un-registering and exiting.

Oct 6 22:42:11 dbs-n2 auditd[4163]: The audit daemon is exiting.

Oct 6 22:42:11 dbs-n2 kernel: audit(1381070531.476:50470): audit_pid=0 old=4163 by auid=4294967295

Oct 6 22:42:11 dbs-n2 pcscd: pcscdaemon.c:572:signal_trap() Preparing for suicide

Oct 6 22:42:12 dbs-n2 pcscd: hotplug_libusb.c:376:HPRescanUsbBus() Hotplug stopped

Oct 6 22:42:12 dbs-n2 pcscd: readerfactory.c:1379:RFCleanupReaders() entering cleaning function

Oct 6 22:42:12 dbs-n2 pcscd: pcscdaemon.c:532:at_exit() cleaning /var/run

Oct 6 22:42:12 dbs-n2 kernel: Kernel logging (proc) stopped.

Oct 6 22:42:12 dbs-n2 kernel: Kernel log daemon terminating.

Oct 6 22:42:13 dbs-n2 exiting on signal 15

Oct 6 22:45:52 dbs-n2 syslogd 1.4.1: restart.

Oct 6 22:45:52 dbs-n2 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Oct 6 22:45:52 dbs-n2 kernel: Bootdata ok (command line is ro root=LABEL=/ rhgb quiet)

Oct 6 22:45:52 dbs-n2 kernel: Linux version 2.6.18-164.el5xen (mockbuild@x86-003.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:59:52 EDT 2009

Oct 6 22:45:52 dbs-n2 kernel: BIOS-provided physical RAM map:

Oct 6 22:45:52 dbs-n2 kernel: Xen: 0000000000000000 - 00000001dc8fa000 (usable)

Oct 6 22:45:52 dbs-n2 kernel: DMI 2.7 present.

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x30] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x34] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x32] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x36] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x31] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x35] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)

Oct 6 22:45:53 dbs-n2 rpc.statd[4213]: Version 1.0.9 Starting

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x23] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x33] enabled)

Oct 6 22:45:53 dbs-n2 rpc.statd[4213]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x27] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x37] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

Oct 6 22:45:53 dbs-n2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])

Oct 6 22:45:53 dbs-n2 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23

Oct 6 22:45:53 dbs-n2 kernel: ACPI: IOAPIC (id[0x00] address[0xfec80000] gsi_base[24])

Oct 6 22:45:53 dbs-n2 kernel: IOAPIC[1]: apic_id 0, version 32, address 0xfec80000, GSI 24-47

Oct 6 22:45:53 dbs-n2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

Oct 6 22:45:53 dbs-n2 kernel: Setting APIC routing to xen

Oct 6 22:45:53 dbs-n2 kernel: Using ACPI (MADT) for SMP configuration information

Oct 6 22:45:53 dbs-n2 kernel: Allocating PCI resources starting at e6000000 (gap: e4000000:1ac00000)

Oct 6 22:45:53 dbs-n2 kernel: Built 1 zonelists. Total pages: 1951994

Oct 6 22:45:53 dbs-n2 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet

Oct 6 22:45:53 dbs-n2 kernel: Initializing CPU#0

Oct 6 22:45:53 dbs-n2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)

Oct 6 22:45:53 dbs-n2 kernel: Xen reported: 2399.364 MHz processor.

Oct 6 22:45:53 dbs-n2 kernel: Console: colour VGA+ 80x25

Oct 6 22:45:54 dbs-n2 kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)

Oct 6 22:45:54 dbs-n2 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)

Oct 6 22:45:54 dbs-n2 kernel: Software IO TLB enabled:

Oct 6 22:45:54 dbs-n2 kernel: Aperture: 64 megabytes

Oct 6 22:45:54 dbs-n2 kernel: Kernel range: 0xffff88000a478000 - 0xffff88000e478000

Oct 6 22:45:54 dbs-n2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

Oct 6 22:45:54 dbs-n2 kernel: Memory: 7569128k/7807976k available (2490k kernel code, 229996k reserved, 1388k data, 184k init)

Oct 6 22:45:54 dbs-n2 hcid[4292]: Bluetooth HCI daemon

Oct 6 22:45:54 dbs-n2 kernel: Calibrating delay using timer specific routine.. 6001.33 BogoMIPS (lpj=12002673)

Oct 6 22:45:54 dbs-n2 kernel: Security Framework v1.0.0 initialized

Oct 6 22:45:54 dbs-n2 kernel: SELinux: Initializing.

Oct 6 22:45:54 dbs-n2 hcid[4292]: Register path:/org/bluez fallback:1

Oct 6 22:45:54 dbs-n2 sdpd[4296]: Bluetooth SDP daemon

Oct 6 22:45:54 dbs-n2 kernel: selinux_register_security: Registering secondary module capability

Oct 6 22:45:54 dbs-n2 kernel: Capability LSM initialized as secondary

Oct 6 22:45:54 dbs-n2 kernel: Mount-cache hash table entries: 256

Oct 6 22:45:54 dbs-n2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

Oct 6 22:45:54 dbs-n2 kernel: CPU: L2 cache: 256K

Oct 6 22:45:54 dbs-n2 kernel: CPU: L3 cache: 12288K

Oct 6 22:45:54 dbs-n2 kernel: CPU: Physical Processor ID: 0

Oct 6 22:45:54 dbs-n2 kernel: CPU: Processor Core ID: 0

Oct 6 22:45:54 dbs-n2 kernel: (SMP-)alternatives turned off

Oct 6 22:45:54 dbs-n2 kernel: ACPI: Core revision 20060707

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#1

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#2

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#3

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#4

Oct 6 22:45:54 dbs-n2 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#5

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#6

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#7

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#8

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#9

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#10

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#11

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#12

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#13

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#14

Oct 6 22:45:55 dbs-n2 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.

Oct 6 22:45:55 dbs-n2 kernel: Brought up 16 CPUs

Oct 6 22:45:55 dbs-n2 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)

Oct 6 22:45:55 dbs-n2 kernel: Initializing CPU#15

Oct 6 22:45:55 dbs-n2 kernel: migration_cost=632

Oct 6 22:45:55 dbs-n2 kernel: checking if image is initramfs... it is

Oct 6 22:45:55 dbs-n2 kernel: Grant table initialized

Oct 6 22:45:55 dbs-n2 kernel: NET: Registered protocol family 16

Oct 6 22:45:55 dbs-n2 kernel: ACPI: bus type pci registered

Oct 6 22:45:55 dbs-n2 kernel: PCI: Not using MMCONFIG.

Oct 6 22:45:55 dbs-n2 kernel: PCI: Using configuration type 1

Oct 6 22:45:55 dbs-n2 kernel: ACPI: Interpreter enabled

Oct 6 22:45:55 dbs-n2 kernel: ACPI: Using IOAPIC for interrupt routing

Oct 6 22:45:55 dbs-n2 kernel: ACPI: No dock devices found.

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Root Bridge (0000:00)

Oct 6 22:45:55 dbs-n2 kernel: PCI: Transparent bridge - 0000:00:1e.0

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 *10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 10 *11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 *7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs *5 7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs *5 7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 10 11) *0, disabled.

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 *10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 *7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay

Oct 6 22:45:55 dbs-n2 kernel: pnp: PnP ACPI init

Oct 6 22:45:55 dbs-n2 kernel: pnp: PnP ACPI: found 11 devices

Oct 6 22:45:55 dbs-n2 kernel: xen_mem: Initialising balloon driver.

Oct 6 22:45:55 dbs-n2 kernel: usbcore: registered new driver usbfs

Oct 6 22:45:55 dbs-n2 kernel: usbcore: registered new driver hub

Oct 6 22:45:55 dbs-n2 kernel: PCI: Using ACPI for IRQ routing

Oct 6 22:45:55 dbs-n2 kernel: PCI: If a device doesnt work, try "pci=routeirq". If it helps, post a report

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: Initializing

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: domain hash size = 128

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: protocols = UNLABELED CIPSOv4

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: unlabeled traffic allowed by default

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x700-0x71f has been reserved

Oct 6 22:45:55 dbs-n2 kernel: PCI: Bridge: 0000:00:01.0

Oct 6 22:45:55 dbs-n2 kernel: IO window: 4000-4fff

Oct 6 22:45:55 dbs-n2 kernel: MEM window: fb700000-fbbfffff

Oct 6 22:45:55 dbs-n2 kernel: PREFETCH window 0x00000000e6000000-0x00000000e60fffff

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:02.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:03.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:04.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:05.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:06.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:07.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:08.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: 5000-5fff

Oct 6 22:45:56 dbs-n2 kernel: MEM window: fbf00000-fbffffff

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window 0x00000000e6100000-0x00000000e61fffff

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:09.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:0a.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.0

Oct 6 22:45:57 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f2000000-f5ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6200000-0x00000000e62fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.2

Oct 6 22:45:57 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f6000000-f9ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6300000-0x00000000e63fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.4

Oct 6 22:45:57 dbs-n2 kernel: IO window: 3000-3fff

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f1d00000-f1ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6400000-0x00000000e64fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1e.0

Oct 6 22:45:57 dbs-n2 kernel: IO window: 2000-2fff

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f1c00000-f1cfffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e8000000-0x00000000efffffff

Oct 6 22:45:57 dbs-n2 hidd[4478]: Bluetooth HID daemon

Oct 6 22:45:57 dbs-n2 kernel: GSI 16 sharing vector 0x98 and IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.2 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.4 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: NET: Registered protocol family 2

Oct 6 22:45:57 dbs-n2 kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP: Hash tables configured (established 262144 bind 65536)

Oct 6 22:45:57 dbs-n2 kernel: TCP reno registered

Oct 6 22:45:57 dbs-n2 kernel: audit: initializing netlink socket (disabled)

Oct 6 22:45:57 dbs-n2 kernel: type=2000 audit(1381070687.472:1): initialized

Oct 6 22:45:57 dbs-n2 kernel: VFS: Disk quotas dquot_6.5.1

Oct 6 22:45:57 dbs-n2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

Oct 6 22:45:57 dbs-n2 kernel: Initializing Cryptographic API

Oct 6 22:45:57 dbs-n2 kernel: alg: No test for crc32c (crc32c-generic)

Oct 6 22:45:57 dbs-n2 kernel: ksign: Installing public key data

Oct 6 22:45:57 dbs-n2 kernel: Loading keyring

Oct 6 22:45:57 dbs-n2 kernel: - Added public key 4BD2703D74512BB

Oct 6 22:45:57 dbs-n2 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)

Oct 6 22:45:57 dbs-n2 kernel: io scheduler noop registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler anticipatory registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler deadline registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler cfq registered (default)

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 last message repeated 9 times

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.2 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.4 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU8 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU4 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU12 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU6 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU14 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU10 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU9 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU5 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU13 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU7 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU15 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU11 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: Thermal Zone (8 C)

Oct 6 22:45:58 dbs-n2 kernel: Real Time Clock Driver v1.12ac

Oct 6 22:45:58 dbs-n2 kernel: hpet_acpi_add: no address or irqs in _CRS

Oct 6 22:45:58 dbs-n2 kernel: Non-volatile memory driver v1.2

Oct 6 22:45:58 dbs-n2 kernel: Linux agpgart interface v0.101 (c) Dave Jones

Oct 6 22:45:58 dbs-n2 kernel: brd: module loaded

Oct 6 22:45:58 dbs-n2 kernel: Xen virtual console successfully installed as ttyS0

Oct 6 22:45:58 dbs-n2 kernel: Event-channel device installed.

Oct 6 22:45:58 dbs-n2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

Oct 6 22:45:58 dbs-n2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

Oct 6 22:45:58 dbs-n2 automount[4518]: lookup_read_master: lookup(nisplus): couldnt locate nis+ table auto.master

Oct 6 22:45:58 dbs-n2 kernel: ide-floppy driver 0.99.newide

Oct 6 22:45:58 dbs-n2 kernel: usbcore: registered new driver hiddev

Oct 6 22:45:58 dbs-n2 kernel: usbcore: registered new driver usbhid

Oct 6 22:45:58 dbs-n2 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver

Oct 6 22:45:58 dbs-n2 kernel: PNP: PS/2 Controller at 0x60,0x64 irq 1,12

Oct 6 22:45:59 dbs-n2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1

Oct 6 22:45:59 dbs-n2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12

Oct 6 22:45:59 dbs-n2 kernel: mice: PS/2 mouse device common for all mice

Oct 6 22:45:59 dbs-n2 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

Oct 6 22:45:59 dbs-n2 kernel: md: bitmap version 4.39

Oct 6 22:45:59 dbs-n2 kernel: TCP bic registered

Oct 6 22:45:59 dbs-n2 kernel: Initializing IPsec netlink socket

Oct 6 22:45:59 dbs-n2 kernel: NET: Registered protocol family 1

Oct 6 22:45:59 dbs-n2 kernel: NET: Registered protocol family 17

Oct 6 22:45:59 dbs-n2 kernel: Initalizing network drop monitor service

Oct 6 22:45:59 dbs-n2 kernel: Write protecting the kernel read-only data: 478k

Oct 6 22:45:59 dbs-n2 kernel: input: AT Translated Set 2 keyboard as /class/input/input0

Oct 6 22:45:59 dbs-n2 kernel: logips2pp: Detected unknown logitech mouse model 0

Oct 6 22:45:59 dbs-n2 kernel: GSI 17 sharing vector 0x49 and IRQ 17

Oct 6 22:45:59 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.7 -> GSI 20 (level, low) -> IRQ 17

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: debug port 1

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: irq 17, io mem 0xf1bf0000

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004

Oct 6 22:45:59 dbs-n2 kernel: usb usb1: configuration #1 chosen from 1 choice

Oct 6 22:45:59 dbs-n2 kernel: hub 1-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 1-0:1.0: 8 ports detected

Oct 6 22:46:00 dbs-n2 kernel: USB Universal Host Controller Interface driver v3.0

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.0 -> GSI 20 (level, low) -> IRQ 17

Oct 6 22:46:00 dbs-n2 logd: : info: logd started with default configuration.

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2

Oct 6 22:46:00 dbs-n2 logd: : info: G_main_add_SignalHandler: Added signal handler for signal 15

Oct 6 22:46:00 dbs-n2 logd: : info: G_main_add_SignalHandler: Added signal handler for signal 15

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: irq 17, io base 0x00001000

Oct 6 22:46:00 dbs-n2 kernel: usb usb2: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 kernel: hub 2-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 2-0:1.0: 2 ports detected

Oct 6 22:46:00 dbs-n2 kernel: input: ImPS/2 Logitech Wheel Mouse as /class/input/input1

Oct 6 22:46:00 dbs-n2 kernel: GSI 18 sharing vector 0x51 and IRQ 18

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.1 -> GSI 23 (level, low) -> IRQ 18

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: irq 18, io base 0x00001020

Oct 6 22:46:00 dbs-n2 kernel: usb usb3: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 kernel: hub 3-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 3-0:1.0: 2 ports detected

Oct 6 22:46:00 dbs-n2 kernel: GSI 19 sharing vector 0x59 and IRQ 19

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.2 -> GSI 22 (level, low) -> IRQ 19

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4

Oct 6 22:46:00 dbs-n2 heartbeat: : info: Version 2 support: false

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: irq 19, io base 0x00001040

Oct 6 22:46:00 dbs-n2 kernel: usb usb4: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 heartbeat: : WARN: Logging daemon is disabled --enabling logging daemon is recommended

Oct 6 22:46:00 dbs-n2 kernel: hub 4-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 heartbeat: : info: **************************

Oct 6 22:46:01 dbs-n2 kernel: hub 4-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 heartbeat: : info: Configuration validated. Starting heartbeat 2.1.4

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.3 -> GSI 23 (level, low) -> IRQ 18

Oct 6 22:46:01 dbs-n2 heartbeat: : info: heartbeat: version 2.1.4

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: UHCI Host Controller

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: irq 18, io base 0x00001060

Oct 6 22:46:01 dbs-n2 kernel: usb usb5: configuration #1 chosen from 1 choice

Oct 6 22:46:01 dbs-n2 kernel: hub 5-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 kernel: hub 5-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 kernel: GSI 20 sharing vector 0x61 and IRQ 20

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:02:00.4 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: UHCI Host Controller

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: new USB bus registered, assigned bus number 6

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: port count misdetected? forcing to 2 ports

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: irq 20, io base 0x00003c00

Oct 6 22:46:01 dbs-n2 kernel: usb usb6: configuration #1 chosen from 1 choice

Oct 6 22:46:01 dbs-n2 kernel: hub 6-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 kernel: hub 6-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 kernel: SCSI subsystem initialized

Oct 6 22:46:01 dbs-n2 kernel: HP CISS Driver (v 3.6.20-RH3)

Oct 6 22:46:01 dbs-n2 kernel: GSI 21 sharing vector 0x69 and IRQ 21

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:05:00.0 -> GSI 28 (level, low) -> IRQ 21

Oct 6 22:46:01 dbs-n2 kernel: cciss0: <0x323a> at PCI 0000:05:00.0 IRQ 240 using DAC
Oct 6 22:46:01 dbs-n2 kernel: blocks= 286677120 block_size= 512
Oct 6 22:46:01 dbs-n2 kernel: heads= 255, sectors= 32, cylinders= 35132
Oct 6 22:46:01 dbs-n2 kernel:
Oct 6 22:46:01 dbs-n2 kernel: blocks= 286677120 block_size= 512
Oct 6 22:46:02 dbs-n2 kernel: heads= 255, sectors= 32, cylinders= 35132
Oct 6 22:46:02 dbs-n2 kernel:
Oct 6 22:46:02 dbs-n2 xinetd[4561]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Oct 6 22:46:02 dbs-n2 kernel: cciss/c0d0: p1 p2 p3 p4 < p5 >
Oct 6 22:46:02 dbs-n2 xinetd[4561]: Started working: 0 available services
Oct 6 22:46:02 dbs-n2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Oct 6 22:46:02 dbs-n2 kernel: QLogic Fibre Channel HBA Driver
Oct 6 22:46:02 dbs-n2 kernel: GSI 22 sharing vector 0x91 and IRQ 22
Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:0b:00.0 -> GSI 31 (level, low) -> IRQ 22

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Found an ISP2432, irq 22, iobase 0xffffc2000002a000

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Configuring PCI space...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Configure NVRAM parameters...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Verifying loaded RISC code...

Oct 6 22:46:02 dbs-n2 kernel: usb 6-1: new full speed USB device using uhci_hcd and address 2

Oct 6 22:46:02 dbs-n2 kernel: usb 6-1: configuration #1 chosen from 1 choice

Oct 6 22:46:02 dbs-n2 kernel: input: HP Virtual Keyboard as /class/input/input2

Oct 6 22:46:02 dbs-n2 kernel: input: USB HID v1.01 Keyboard on usb-0000:02:00.4-1

Oct 6 22:46:02 dbs-n2 kernel: input: HP Virtual Keyboard as /class/input/input3

Oct 6 22:46:02 dbs-n2 kernel: input: USB HID v1.01 Mouse on usb-0000:02:00.4-1

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Allocated (64 KB) for EFT...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Allocated (1413 KB) for firmware dump...

Oct 6 22:46:02 dbs-n2 kernel: scsi0 : qla2xxx

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0:

Oct 6 22:46:02 dbs-n2 kernel: QLogic Fibre Channel HBA Driver: 8.03.00.10.05.04-k

Oct 6 22:46:02 dbs-n2 kernel: QLogic HPAE311A - PCI-Express 4Gb Fibre Channel HBA

Oct 6 22:46:02 dbs-n2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0b:00.0 hdma+, host#=0, fw=4.04.09 (486)

Oct 6 22:46:02 dbs-n2 kernel: Initializing USB Mass Storage driver...

Oct 6 22:46:02 dbs-n2 kernel: usbcore: registered new driver usb-storage

Oct 6 22:46:02 dbs-n2 kernel: USB Mass Storage support registered.

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: uevent: version 1.0.3

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: dm-raid45: initialized v0.2594l

Oct 6 22:46:02 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:02 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:02 dbs-n2 kernel: SELinux: Disabled at runtime.

Oct 6 22:46:02 dbs-n2 kernel: type=1404 audit(1381070718.142:2): selinux=0 auid=4294967295 ses=4294967295

Oct 6 22:46:02 dbs-n2 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:03:00.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:46:02 dbs-n2 kernel: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 16, node addr 68b599787f22

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:03:00.1 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:02 dbs-n2 kernel: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 20, node addr 68b599787f24

Oct 6 22:46:02 dbs-n2 kernel: GSI 23 sharing vector 0x99 and IRQ 23

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:04:00.0 -> GSI 18 (level, low) -> IRQ 23

Oct 6 22:46:02 dbs-n2 kernel: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 23, node addr 68b599787f26

Oct 6 22:46:02 dbs-n2 kernel: GSI 24 sharing vector 0xA1 and IRQ 24

Oct 6 22:46:03 dbs-n2 kernel: ACPI: PCI Interrupt 0000:04:00.1 -> GSI 19 (level, low) -> IRQ 24

Oct 6 22:46:03 dbs-n2 kernel: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 24, node addr 68b599787f28

Oct 6 22:46:03 dbs-n2 kernel: input: PC Speaker as /class/input/input4

Oct 6 22:46:03 dbs-n2 kernel: ACPI: PCI Interrupt 0000:02:00.2 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:03 dbs-n2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

Oct 6 22:46:03 dbs-n2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

Oct 6 22:46:03 dbs-n2 kernel: Floppy drive(s): fd0 is 1.44M

Oct 6 22:46:03 dbs-n2 kernel: floppy0: no floppy controllers found

Oct 6 22:46:03 dbs-n2 kernel: Floppy drive(s): fd0 is 1.44M

Oct 6 22:46:03 dbs-n2 heartbeat: : info: Heartbeat generation: 1292541790

Oct 6 22:46:03 dbs-n2 kernel: floppy0: no floppy controllers found

Oct 6 22:46:03 dbs-n2 kernel: lp: driver loaded but no devices found

Oct 6 22:46:03 dbs-n2 kernel: NET: Registered protocol family 10

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1

Oct 6 22:46:03 dbs-n2 kernel: lo: Disabled Privacy Extensions

Oct 6 22:46:03 dbs-n2 kernel: IPv6 over IPv4 tunneling driver

Oct 6 22:46:03 dbs-n2 kernel: ACPI: Power Button (FF)

Oct 6 22:46:03 dbs-n2 kernel: md: Autodetecting RAID arrays.

Oct 6 22:46:03 dbs-n2 kernel: md: autorun ...

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1

Oct 6 22:46:03 dbs-n2 kernel: md: ... autorun DONE.

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: ping heartbeat started.

Oct 6 22:46:03 dbs-n2 kernel: device-mapper: multipath: version 1.0.5 loaded

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p2, internal journal

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler

Oct 6 22:46:03 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_SignalHandler: Added signal handler for signal 17

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p5, internal journal

Oct 6 22:46:03 dbs-n2 heartbeat: : info: Local status now set to: 'up'

Oct 6 22:46:03 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:03 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p1, internal journal

Oct 6 22:46:03 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:03 dbs-n2 kernel: Adding 16386292k swap on /dev/cciss/c0d0p3. Priority:-1 extents:1 across:16386292k

Oct 6 22:46:03 dbs-n2 kernel: IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com>

Oct 6 22:46:03 dbs-n2 kernel: Loading iSCSI transport class v2.0-871.

Oct 6 22:46:03 dbs-n2 kernel: 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>

Oct 6 22:46:03 dbs-n2 kernel: All bugs added by David S. Miller <davem@redhat.com>

Oct 6 22:46:03 dbs-n2 kernel: cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.

Oct 6 22:46:03 dbs-n2 kernel: iscsi: registered transport (cxgb3i)

Oct 6 22:46:04 dbs-n2 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.0.0 (March 21, 2009)

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth0

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth1

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth2

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth3

Oct 6 22:46:04 dbs-n2 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.0.1e (June 22, 2009)

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (bnx2i)

Oct 6 22:46:04 dbs-n2 kernel: scsi1 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi2 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi3 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi4 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (tcp)

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (iser)

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth0: using MSIX

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth1: using MSIX

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth1

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth1

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

Oct 6 22:46:04 dbs-n2 kernel: device-mapper: dm-log-clustered: (built Jul 27 2009 16:28:42) installed

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: Core ver 2.10

Oct 6 22:46:04 dbs-n2 kernel: NET: Registered protocol family 31

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HCI device and connection manager initialized

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HCI socket layer initialized

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: L2CAP ver 2.8

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Link dbs-n1:eth1 up.

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: L2CAP socket layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Status update for node dbs-n1: status up

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM socket layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 60 ms (> 50 ms) (GSource: 0x101b5858)

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM TTY layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 up.

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM ver 1.8

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Status update for node 192.191.0.167: status ping

Oct 6 22:46:04 dbs-n2 harc[4690]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1

Oct 6 22:46:04 dbs-n2 heartbeat: : WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 120 ms (> 50 ms) (GSource: 0x101b8c18)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Link dbs-n2:eth1 up.

Oct 6 22:46:05 dbs-n2 heartbeat: : WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 110 ms (> 100 ms) before being called (GSource: 0x101b8458)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: G_SIG_dispatch: started at 1718072208 should have started at 1718072197

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Comm_now_up(): updating status to active

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Local status now set to: 'active'

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Starting child client "/usr/lib64/heartbeat/ipfail" (201,204)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Status update for node dbs-n1: status active

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Starting "/usr/lib64/heartbeat/ipfail" as uid 201 gid 204 (pid 4707)

Oct 6 22:46:05 dbs-n2 harc[4708]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:46:13 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: local resource transition completed.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: Initial resource acquisition complete (T_RESOURCES(us))

Oct 6 22:46:16 dbs-n2 heartbeat: : info: No local resources to acquire.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:46:59 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:46:59 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:46:59 dbs-n2 heartbeat: : info: acquire local HA resources (standby).

Oct 6 22:46:59 dbs-n2 heartbeat: : info: local HA resource acquisition completed (standby).

Oct 6 22:46:59 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:47:00 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:49:08 dbs-n2 ipfail: : info: Telling other node that we have more visible ping nodes.

Oct 6 22:49:13 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:49:13 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:49:25 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:49:25 dbs-n2 heartbeat: : info: acquire all HA resources (standby).

Oct 6 22:49:25 dbs-n2 ResourceManager[4812]: info: Acquiring resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:49:25 dbs-n2 IPaddr[4839]: INFO: Resource is stopped

Oct 6 22:49:25 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 start

Oct 6 22:49:26 dbs-n2 IPaddr[4937]: INFO: Using calculated netmask for 192.191.0.22: 255.255.255.0

Oct 6 22:49:26 dbs-n2 IPaddr[4937]: INFO: eval ifconfig eth0:0 192.191.0.22 netmask 255.255.255.0 broadcast 192.191.0.255

Oct 6 22:49:26 dbs-n2 kernel: send_arp uses obsolete (PF_INET,SOCK_PACKET)

Oct 6 22:49:26 dbs-n2 IPaddr[4908]: INFO: Success

Oct 6 22:49:26 dbs-n2 Filesystem[5055]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:26 dbs-n2 Filesystem[5044]: INFO: Resource is stopped

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 start

Oct 6 22:49:26 dbs-n2 Filesystem[5130]: INFO: Running start for /dev/sda1 on /clusdsk

Oct 6 22:49:26 dbs-n2 Filesystem[5130]: ERROR: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:26 dbs-n2 Filesystem[5119]: ERROR: Illegal argument

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: ERROR: Return code 2 from /etc/ha.d/resource.d/Filesystem

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: CRIT: Giving up resources due to failure of Filesystem::/dev/sda1::/clusdsk::ext3

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:30 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:30 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:31 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: WARN: Machine reboot narrowly avoided!

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:49:37 dbs-n2 Filesystem[5601]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:37 dbs-n2 Filesystem[5601]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:49:37 dbs-n2 Filesystem[5590]: INFO: Success

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:49:37 dbs-n2 IPaddr[5707]: INFO: ifconfig eth0:0 down

Oct 6 22:49:37 dbs-n2 IPaddr[5678]: INFO: Success

Oct 6 22:49:37 dbs-n2 heartbeat: : info: all HA resource acquisition completed (standby).

Oct 6 22:49:37 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:49:38 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:49:46 dbs-n2 gpm[5749]: *** info :

Oct 6 22:49:46 dbs-n2 gpm[5749]: Started gpm successfully. Entered daemon mode.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully dropped root privileges.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: avahi-daemon 0.6.16 starting up.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully called chroot().

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully dropped remaining capabilities.

Oct 6 22:49:47 dbs-n2 kernel: Bridge firewalling registered

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Loading service file /services/sftp-ssh.service.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth1.IPv6 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::6ab5:99ff:fe78:7f24.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth1.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth1.IPv4 with address 10.9.0.21.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv6 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Network interface enumeration completed.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f24 on eth1.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for 10.9.0.21 on eth1.

Oct 6 22:49:47 dbs-n2 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.191.0.21 on eth0.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering HINFO record with values 'X86_64'/'LINUX'.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface virbr0.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 kernel: Netfilter messages via NETLINK v0.30.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.

Oct 6 22:49:48 dbs-n2 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.168.122.1 on virbr0.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: New relevant interface virbr0.IPv6 for mDNS.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::200:ff:fe00:0 on virbr0.

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: started, version 2.45 cachesize 150

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: compile time options: IPv6 GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: reading /etc/resolv.conf

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: using nameserver 192.191.0.20#53

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: read /etc/hosts - 6 addresses

Oct 6 22:49:49 dbs-n2 avahi-daemon[5871]: Server startup complete. Host name is dbs-n2.local. Local service cookie is 287356982.

Oct 6 22:49:50 dbs-n2 avahi-daemon[5871]: Service "SFTP File Transfer on dbs-n2" (/services/sftp-ssh.service) successfully established.

Oct 6 22:49:51 dbs-n2 xenstored: Checking store ...

Oct 6 22:49:52 dbs-n2 xenstored: Checking store complete.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Withdrawing address record for 192.191.0.21 on eth0.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: iface.c: interface_mdns_mcast_join() called but no local address available.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Interface eth0.IPv4 no longer relevant for mDNS.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Withdrawing address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: iface.c: interface_mdns_mcast_join() called but no local address available.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Interface eth0.IPv6 no longer relevant for mDNS.

Oct 6 22:49:52 dbs-n2 kernel: device vif0.0 entered promiscuous mode

Oct 6 22:49:52 dbs-n2 kernel: xenbr0: topology change detected, propagating

Oct 6 22:49:52 dbs-n2 kernel: xenbr0: port 1(vif0.0) entering forwarding state

Oct 6 22:49:52 dbs-n2 kernel: bnx2: peth0: using MSIX

Oct 6 22:49:52 dbs-n2 kernel: ADDRCONF(NETDEV_UP): peth0: link is not ready

Oct 6 22:49:52 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=peth0

Oct 6 22:49:53 dbs-n2 heartbeat: : ERROR: glib: Error sending packet: Network is unreachable

Oct 6 22:49:53 dbs-n2 heartbeat: : info: glib: euid=99 egid=99

Oct 6 22:49:53 dbs-n2 heartbeat: : ERROR: write_child: write failure on ping 192.191.0.167.: Network is unreachable

Oct 6 22:49:54 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=peth0

Oct 6 22:49:55 dbs-n2 heartbeat: : ERROR: glib: Error sending packet: Network is unreachable

Oct 6 22:49:55 dbs-n2 heartbeat: : info: glib: euid=99 egid=99

Oct 6 22:49:55 dbs-n2 heartbeat: : ERROR: write_child: write failure on ping 192.191.0.167.: Network is unreachable

Oct 6 22:49:55 dbs-n2 kernel: bnx2: peth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:49:55 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): peth0: link becomes ready

Oct 6 22:49:56 dbs-n2 kernel: device peth0 entered promiscuous mode

Oct 6 22:49:56 dbs-n2 kernel: xenbr0: topology change detected, propagating

Oct 6 22:49:56 dbs-n2 kernel: xenbr0: port 2(peth0) entering forwarding state

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv4 for mDNS.

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.191.0.21 on eth0.

Oct 6 22:49:57 dbs-n2 heartbeat: : WARN: node 192.191.0.167: is dead

Oct 6 22:49:57 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 dead.

Oct 6 22:49:57 dbs-n2 ipfail: : info: Status update: Node 192.191.0.167 now has status dead

Oct 6 22:49:57 dbs-n2 harc[6780]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:49:57 dbs-n2 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).

Oct 6 22:49:57 dbs-n2 kernel: Hangcheck: Using monotonic_clock().

Oct 6 22:49:58 dbs-n2 modclusterd: startup succeeded

Oct 6 22:49:58 dbs-n2 ipfail: : info: NS: We are dead. :<

Oct 6 22:49:58 dbs-n2 ipfail: : info: Link Status update: Link 192.191.0.167/192.191.0.167 now has status dead

Oct 6 22:49:58 dbs-n2 oddjobd: oddjobd startup succeeded

Oct 6 22:49:58 dbs-n2 saslauthd[6953]: detach_tty : master pid is: 6953

Oct 6 22:49:58 dbs-n2 saslauthd[6953]: ipc_init : listening on socket: /var/run/saslauthd/mux

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv6 for mDNS.

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:58 dbs-n2 ricci: startup succeeded

Oct 6 22:49:58 dbs-n2 smartd[6976]: smartd version 5.38 Copyright (C) 2002-8 Bruce Allen

Oct 6 22:49:58 dbs-n2 smartd[6976]: Home page is http://smartmontools.sourceforge.net/

Oct 6 22:49:58 dbs-n2 smartd[6976]: Opened configuration file /etc/smartd.conf

Oct 6 22:49:58 dbs-n2 smartd[6976]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices

Oct 6 22:49:58 dbs-n2 smartd[6976]: Problem creating device name scan list

Oct 6 22:49:58 dbs-n2 smartd[6976]: Problem creating device name scan list

Oct 6 22:49:58 dbs-n2 smartd[6976]: In the system's table of devices NO devices found to scan

Oct 6 22:49:58 dbs-n2 smartd[6976]: Monitoring 0 ATA and 0 SCSI devices

Oct 6 22:49:58 dbs-n2 smartd[6978]: smartd has fork()ed into background mode. New PID=6978.

Oct 6 22:49:59 dbs-n2 ipfail: : info: We are dead. :<

Oct 6 22:49:59 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:50:00 dbs-n2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found

Oct 6 22:50:00 dbs-n2 last message repeated 3 times

Oct 6 22:50:00 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 up.

Oct 6 22:50:00 dbs-n2 heartbeat: : info: Status update for node 192.191.0.167: status ping

Oct 6 22:50:00 dbs-n2 ipfail: : info: Link Status update: Link 192.191.0.167/192.191.0.167 now has status up

Oct 6 22:50:00 dbs-n2 ipfail: : info: Status update: Node 192.191.0.167 now has status ping

Oct 6 22:50:00 dbs-n2 ipfail: : info: A ping node just came up.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Giving up because we were told that we have less ping nodes.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Delayed giveup in 4 seconds.

Oct 6 22:50:04 dbs-n2 ipfail: : info: Aborted delayed giveup (3)

Oct 6 22:50:07 dbs-n2 hb_standby[7146]: Going standby .

Oct 6 22:50:08 dbs-n2 heartbeat: : info: dbs-n2 wants to go standby

Oct 6 22:50:08 dbs-n2 heartbeat: : info: standby: dbs-n1 can take our foreign resources

Oct 6 22:50:08 dbs-n2 heartbeat: : info: give up foreign HA resources (standby).

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:11 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:12 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:12 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: WARN: Machine reboot narrowly avoided!

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:50:19 dbs-n2 Filesystem[7590]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:50:19 dbs-n2 Filesystem[7590]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:50:19 dbs-n2 Filesystem[7579]: INFO: Success

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:50:19 dbs-n2 IPaddr[7667]: INFO: Success

Oct 6 22:50:19 dbs-n2 heartbeat: : info: foreign HA resource release completed (standby).

Oct 6 22:50:19 dbs-n2 heartbeat: : info: Local standby process completed .

Oct 6 22:50:32 dbs-n2 heartbeat: : WARN: 1 lost packet(s) for

Oct 6 22:50:32 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:50:32 dbs-n2 heartbeat: : info: No pkts missing from dbs-n1!

Oct 6 22:50:32 dbs-n2 heartbeat: : info: Other node completed standby takeover of foreign resources.

Oct 6 22:50:45 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:50:45 dbs-n2 ipfail: : info: Giving up foreign resources (auto_failback).

Oct 6 22:50:45 dbs-n2 ipfail: : info: Delayed giveup in 4 seconds.

Oct 6 22:50:49 dbs-n2 ipfail: : info: giveup() called (timeout worked)

Oct 6 22:50:49 dbs-n2 heartbeat: : info: dbs-n2 wants to go standby

Oct 6 22:50:50 dbs-n2 heartbeat: : info: standby: dbs-n1 can take our foreign resources

Oct 6 22:50:50 dbs-n2 heartbeat: : info: give up foreign HA resources (standby).

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: WARN: Machine reboot narrowly avoided!

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:51:01 dbs-n2 Filesystem[8149]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:51:01 dbs-n2 Filesystem[8149]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:51:01 dbs-n2 Filesystem[8138]: INFO: Success

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:51:01 dbs-n2 IPaddr[8226]: INFO: Success

Oct 6 22:51:01 dbs-n2 heartbeat: : info: foreign HA resource release completed (standby).

Oct 6 22:51:01 dbs-n2 heartbeat: : info: Local standby process completed .

Oct 6 22:51:03 dbs-n2 heartbeat: : WARN: Standby in progress- new request from dbs-n1 ignored

Oct 6 22:51:14 dbs-n2 heartbeat: : WARN: 1 lost packet(s) for

Oct 6 22:51:14 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:51:14 dbs-n2 heartbeat: : info: No pkts missing from dbs-n1!

Oct 6 22:51:14 dbs-n2 heartbeat: : info: Other node completed standby takeover of foreign resources.

Oct 6 22:51:44 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:51:45 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:51:45 dbs-n2 heartbeat: : info: acquire local HA resources (standby).

Oct 6 22:51:45 dbs-n2 heartbeat: : info: local HA resource acquisition completed (standby).

Oct 6 22:51:45 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:51:45 dbs-n2 heartbeat: : info: remote resource transition completed.

#

大家能帮我分析问题出在哪里,及如何确认和排查吗?

wang.hy8166

0 项奖励
20 回复数
Highlighted
5 Tungsten

Re: 存储突然链接失败,请教如何查找原因?

您的这个Oracle的存储规划也太那个了吧,

您可以试试看这样的顺序启动Oracle看看在那个步骤卡了。

startup nomount

alter database mount

alter database open

您的OS是啥系统呢?使用了EMC的存储了么?用了powerpath了么? 如果有用powermt display看看

还有就是找人上存储和SAN switch上看看是不是硬件和链路有问题

在没有得到具体信息之前不要尝试使用fsck和一些verify工具修复数据文件

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

非常感谢您的回复!

还是直接找厂商的人来看。

wang.hy8166

发件人: lei liu

发送时间: 2013-10-07 08:34

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from lei liu in 综合讨论区 - View the full discussion

您的这个Oracle的存储规划也太那个了吧,

您可以试试看这样的顺序启动Oracle看看在那个步骤卡了。

startup nomount

alter database mount

alter database open

您的OS是啥系统呢?使用了EMC的存储了么?用了powerpath了么? 如果有用powermt display看看

还有就是找人上存储和SAN switch上看看是不是硬件和链路有问题

在没有得到具体信息之前不要尝试使用fsck和一些verify工具修复数据文件

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
4 Beryllium

Re: 存储突然链接失败,请教如何查找原因?

恩,还在假期,坛子里人少。建议打技术热线电话找原厂工程师来看看

0 项奖励
Highlighted
4 Beryllium

Re: 存储突然链接失败,请教如何查找原因?

lz,找到问题根源了不?如果方便的话,可否把后继解决方法给大家分享下,呵呵

0 项奖励
Highlighted
5 Tungsten

Re: 存储突然链接失败,请教如何查找原因?

这兄弟的环境看样子好像是RAC双节点的环境,不然主机都这样了,Oracle早就罢工了

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

是的,oracle罢工了,找厂商过来

wang.hy8166

发件人: lei liu

发送时间: 2013-10-18 19:35

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from lei liu in 综合讨论区 - View the full discussion

这兄弟的环境看样子好像是RAC双节点的环境,不然主机都这样了,Oracle早就罢工了

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

先关闭2台主机,再关闭存储,10分钟后,开启存储,再开主机,正常

然后找厂商过来

wang.hy8166

发件人: Yanhong Huang

发送时间: 2013-10-18 10:21

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from Yanhong Huang in 综合讨论区 - View the full discussion

lz,找到问题根源了不?如果方便的话,可否把后继解决方法给大家分享下,呵呵

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
3 Silver

Re: 存储突然链接失败,请教如何查找原因?

一般情况下,如果通过重启服务器能解决的问题,极少会是存储问题。

另外,一个基本经验是:有存储故障也不要重启存储来解决问题!重启存储一般只会把问题搞得更复杂,会破坏现场环境,把修复变得更困难,严重时数据丢失!

数据丢失是存储最严重的事故。

存储故障一定要找到原因,针对原因修复故障!不要轻易重启!

Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

谢谢提醒,下次注意了。

wang.hy8166

发件人: Jianyun

发送时间: 2013-10-25 13:56

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from Jianyun in 综合讨论区 - View the full discussion

一般情况下,如果通过重启服务器能解决的问题,极少会是存储问题。

另外,一个基本经验是:有存储故障也不要重启存储来解决问题!重启存储一般只会把问题搞得更复杂,会破坏现场环境,把修复变得更困难,严重时数据丢失!

数据丢失是存储最严重的事故。

存储故障一定要找到原因,针对原因修复故障!不要轻易重启!

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

--

0 项奖励
Highlighted
5 Tungsten

Re: 存储突然链接失败,请教如何查找原因?

一般系统都会有“句柄”,即使存储中的数据没有了但“句柄”还在运气好还能把丢失的文件找回来,重新启动OS那就啥都没有了。

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

嗯,看来我得恶补下了。

您有相关的案例文档供学习吗?

wang.hy8166

发件人: lei liu

发送时间: 2013-10-25 14:59

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from lei liu in 综合讨论区 - View the full discussion

一般系统都会有“句柄”,即使存储中的数据没有了但“句柄”还在运气好还能把丢失的文件找回来,重新启动OS那就啥都没有了。

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
5 Tungsten

Re: 存储突然链接失败,请教如何查找原因?

0 项奖励
Highlighted
4 Beryllium

Re: 存储突然链接失败,请教如何查找原因?

控制文件和日志文件都存放在'/clusdsk/oracle/oradata/ORCL上面,发生Linux-x86_64 Error: 30: Read-only file system,就是/dev/sda1             825G  164G  619G  21% /clusdsk这个文件系统。

OS上看到试图访问时的错误代码:sd 0:0:0:0: SCSI error: return code = 0x00010000

可能是路径不可用或者阵列物理故障。不了解LZ的环境,不好下判断。

0 项奖励
Highlighted
5 Tungsten

Re: 存储突然链接失败,请教如何查找原因?

楼主 ,这个问题后来是怎么解决的啊。

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

先关闭2台主机,再关闭存储,10分钟后,开启存储,再开主机,正常

然后找厂商过来

wang.hy8166

发件人: lei liu

发送时间: 2013-10-18 19:36

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from lei liu in 综合讨论区 - View the full discussion

楼主 ,这个问题后来是怎么解决的啊。

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

Highlighted
2 Iron

Re: 存储突然链接失败,请教如何查找原因?

节点有问题

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

是存储设备有问题

wang.hy8166

发件人: 独孤月

发送时间: 2013-10-20 13:54

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from 独孤月 in 综合讨论区 - View the full discussion

节点有问题

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
3 Zinc

Re: 存储突然链接失败,请教如何查找原因?

还是找厂商吧,确认一下,这个实际环境都不一样的。

0 项奖励
Highlighted
2 Bronze

Re: 存储突然链接失败,请教如何查找原因?

是的,最后找了厂商过来。

wang.hy8166

发件人: cxemc

发送时间: 2013-10-20 14:20

收件人: hy8166

主题: Re: - 存储突然链接失败,请教如何查找原因?

ECN

存储突然链接失败,请教如何查找原因?

reply from cxemc in 综合讨论区 - View the full discussion

还是找厂商吧,确认一下,这个实际环境都不一样的。

Reply to this message by replying to this email, or go to the message on ECN

Start a new discussion in 综合讨论区 by email or at ECN

Following 存储突然链接失败,请教如何查找原因? in these streams: Inbox

0 项奖励
Highlighted
3 Zinc

Re: 存储突然链接失败,请教如何查找原因?

硬件上去排查吧这个问题。

0 项奖励