存储突然链接失败，请教如何查找原因？

各位好！

今天发现这样一个问题，我的同事反馈使用PL/SQL Developer登录oracle数据库失败，然后我登录服务器发现oracle实例未启动，于是尝试启动

SQL> startup

ORACLE instance started.

Total System Global Area 4764729344 bytes

Fixed Size 2027160 bytes

Variable Size 419430760 bytes

Database Buffers 4328521728 bytes

Redo Buffers 14749696 bytes

ORA-00205: error in identifying control file, check alert log for more info

SQL>

根据提示，我查看alert日志

Sun Oct 6 21:08:37 2013

Thread 1 advanced to log sequence 357916

Current log# 3 seq# 357916 mem# 0: /clusdsk/oracle/oradata/ORCL/redo03.log

Sun Oct 6 21:26:27 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_lgwr_17049.trc:

ORA-00345: redo log write error block 41385 count 2

ORA-00312: online log 3 thread 1: '/clusdsk/oracle/oradata/ORCL/redo03.log'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 41385

Additional information: -1

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_ckpt_17051.trc:

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control03.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control02.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control01.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_ckpt_17051.trc:

ORA-00221: error on write to control file

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control03.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control02.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/clusdsk/oracle/oradata/ORCL/control01.ctl'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 3

Additional information: -1

Sun Oct 6 21:26:28 2013

KCF: write/open error block=0x29497 online=1

file=19 /clusdsk/oracle/oradata/ORCL/TBS12.dbf

error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 169111

Additional information: -1'

Sun Oct 6 21:26:28 2013

KCF: write/open error block=0x294a7 online=1

file=19 /clusdsk/oracle/oradata/ORCL/TBS12.dbf

error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 169127

Additional information: -1'

Sun Oct 6 21:26:28 2013

CKPT: terminating instance due to error 221

Sun Oct 6 21:26:28 2013

Automatic datafile offline due to write error on

file 19: /clusdsk/oracle/oradata/ORCL/TBS12.dbf

Sun Oct 6 21:26:28 2013

Automatic datafile offline due to write error on

file 19: /clusdsk/oracle/oradata/ORCL/TBS12.dbf

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_dbw1_17047.trc:

ORA-00221: error on write to control file

Sun Oct 6 21:26:28 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_dbw0_17045.trc:

ORA-00221: error on write to control file

Sun Oct 6 21:26:29 2013

Errors in file /opt/oracle/admin/ORCL/bdump/ORCL_lgwr_17049.trc:

ORA-00340: IO error processing online log 3 of thread 1

ORA-00345: redo log write error block 41385 count 2

ORA-00312: online log 3 thread 1: '/clusdsk/oracle/oradata/ORCL/redo03.log'

ORA-27072: File I/O error

Linux-x86_64 Error: 30: Read-only file system

Additional information: 4

Additional information: 41385

Additional information: -1

Sun Oct 6 21:26:34 2013

Instance terminated by CKPT, pid = 17051

然后，我感觉是和存储链路有关

于是，访问磁盘试试看

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 29G 5.5G 22G 21% /

/dev/cciss/c0d0p5 89G 5.1G 80G 7% /home

/dev/cciss/c0d0p1 99M 20M 75M 21% /boot

tmpfs 3.8G 0 3.8G 0% /dev/shm

none 3.8G 104K 3.8G 1% /var/lib/xenstored

/dev/sda1 825G 164G 619G 21% /clusdsk

$ cd /clusdsk

$ ls

ls: reading directory .: Input/output error

$

接着，我又查看了系统日志

# cat messages

Oct 6 04:03:24 dbs-n2 syslogd 1.4.1: restart.

Oct 6 21:26:11 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP DOWN detected (2 6 0).

Oct 6 21:26:27 dbs-n2 kernel: rport-0:0-0: blocked FC remote port time out: saving binding

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 214919

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 396623951

Oct 6 21:26:27 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 49577986

Oct 6 21:26:27 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:27 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:27 dbs-n2 kernel: end_request: I/O error, dev sda, sector 398196815

Oct 6 21:26:27 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 49774594

Oct 6 21:26:27 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:27 dbs-n2 kernel: Aborting journal on device sda1.

Oct 6 21:26:27 dbs-n2 kernel: EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted

Oct 6 21:26:27 dbs-n2 last message repeated 2 times

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12423

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 1545

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 63

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 0

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: EXT3-fs error (device sda1) in ext3_dirty_inode: Journal has aborted

Oct 6 21:26:28 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 21:26:28 dbs-n2 kernel: end_request: I/O error, dev sda, sector 63

Oct 6 21:26:28 dbs-n2 kernel: Buffer I/O error on device sda1, logical block 0

Oct 6 21:26:28 dbs-n2 kernel: lost page write due to I/O error on sda1

Oct 6 21:26:28 dbs-n2 kernel: ext3_abort called.

Oct 6 21:26:28 dbs-n2 kernel: EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal

Oct 6 21:26:28 dbs-n2 kernel: Remounting filesystem read-only

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LIP reset occured (f8f7).

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LIP occured (f8f7).

Oct 6 21:43:09 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP UP detected (4 Gbps).

Oct 6 21:48:04 dbs-n2 kernel: qla2xxx 0000:0b:00.0: LOOP DOWN detected (2 6 0).

Oct 6 21:48:20 dbs-n2 kernel: rport-0:0-0: blocked FC remote port time out: saving binding

Oct 6 22:33:44 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:33:44 dbs-n2 kernel: end_request: I/O error, dev sda, sector 1050153047

Oct 6 22:33:44 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:33:44 dbs-n2 kernel: end_request: I/O error, dev sda, sector 1050153047

Oct 6 22:38:31 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:38:31 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12375

Oct 6 22:39:33 dbs-n2 kernel: sd 0:0:0:0: SCSI error: return code = 0x00010000

Oct 6 22:39:33 dbs-n2 kernel: end_request: I/O error, dev sda, sector 12375

Oct 6 22:41:44 dbs-n2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found

Oct 6 22:41:47 dbs-n2 modclusterd: shutdown succeeded

Oct 6 22:41:48 dbs-n2 ricci: shutdown succeeded

Oct 6 22:41:49 dbs-n2 smartd[6631]: smartd received signal 15: Terminated

Oct 6 22:41:49 dbs-n2 smartd[6631]: smartd is exiting (exit status 0)

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Got SIGTERM, quitting.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth1.IPv6 with address fe80::6ab5:99ff:fe78:7f24.

Oct 6 22:41:49 dbs-n2 avahi-daemon[5524]: Leaving mDNS multicast group on interface eth1.IPv4 with address 10.9.0.21.

Oct 6 22:41:49 dbs-n2 oddjobd: oddjobd shutdown succeeded

Oct 6 22:41:50 dbs-n2 libvirtd: 22:41:50.025: warning : Shutting down on signal 15

Oct 6 22:41:50 dbs-n2 heartbeat: : WARN: Shutdown delayed until current resource activity finishes.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Heartbeat shutdown in progress. (4753)

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Received shutdown notice from 'dbs-n1'.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Resource takeover cancelled - shutdown in progress.

Oct 6 22:41:56 dbs-n2 heartbeat: : info: Giving up all HA resources.

Oct 6 22:41:56 dbs-n2 ResourceManager[9308]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:41:56 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:42:00 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: Trying to unmount /clusdsk

Oct 6 22:42:01 dbs-n2 kernel: ext3_abort called.

Oct 6 22:42:01 dbs-n2 kernel: EXT3-fs error (device sda1): ext3_put_super: Couldnt clean up the journal

Oct 6 22:42:01 dbs-n2 Filesystem[9469]: INFO: unmounted /clusdsk successfully

Oct 6 22:42:01 dbs-n2 Filesystem[9458]: INFO: Success

Oct 6 22:42:01 dbs-n2 ResourceManager[9308]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:42:01 dbs-n2 IPaddr[9588]: INFO: ifconfig eth0:0 down

Oct 6 22:42:01 dbs-n2 IPaddr[9559]: INFO: Success

Oct 6 22:42:01 dbs-n2 heartbeat: : info: All HA resources relinquished.

Oct 6 22:42:02 dbs-n2 heartbeat: : info: Link dbs-n1:eth1 dead.

Oct 6 22:42:02 dbs-n2 ipfail: : info: Link Status update: Link dbs-n1/eth1 now has status dead

Oct 6 22:42:02 dbs-n2 heartbeat: : info: killing /usr/lib64/heartbeat/ipfail process group 4791 with signal 15

Oct 6 22:42:03 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:42:03 dbs-n2 ipfail: : info: Checking remote count of ping nodes.

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBFIFO process 4768 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBWRITE process 4769 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBREAD process 4770 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBWRITE process 4771 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: killing HBREAD process 4772 with signal 15

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4768 exited. 5 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4769 exited. 4 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4771 exited. 3 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4772 exited. 2 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: Core process 4770 exited. 1 remaining

Oct 6 22:42:04 dbs-n2 heartbeat: : info: dbs-n2 Heartbeat shutdown complete.

Oct 6 22:42:04 dbs-n2 logd: : info: Waiting for pid=4660 to exit

Oct 6 22:42:04 dbs-n2 logd: : info: logd_term_write_action: received SIGTERM

Oct 6 22:42:04 dbs-n2 logd: : info: Exiting write process

Oct 6 22:42:05 dbs-n2 logd: : info: Pid 4660 exited

Oct 6 22:42:05 dbs-n2 saslauthd[6606]: server_exit : master exited: 6606

Oct 6 22:42:06 dbs-n2 xinetd[4645]: Exiting...

Oct 6 22:42:10 dbs-n2 hcid[4374]: Got disconnected from the system message bus

Oct 6 22:42:11 dbs-n2 rpc.statd[4295]: Caught signal 15, un-registering and exiting.

Oct 6 22:42:11 dbs-n2 auditd[4163]: The audit daemon is exiting.

Oct 6 22:42:11 dbs-n2 kernel: audit(1381070531.476:50470): audit_pid=0 old=4163 by auid=4294967295

Oct 6 22:42:11 dbs-n2 pcscd: pcscdaemon.c:572:signal_trap() Preparing for suicide

Oct 6 22:42:12 dbs-n2 pcscd: hotplug_libusb.c:376:HPRescanUsbBus() Hotplug stopped

Oct 6 22:42:12 dbs-n2 pcscd: readerfactory.c:1379:RFCleanupReaders() entering cleaning function

Oct 6 22:42:12 dbs-n2 pcscd: pcscdaemon.c:532:at_exit() cleaning /var/run

Oct 6 22:42:12 dbs-n2 kernel: Kernel logging (proc) stopped.

Oct 6 22:42:12 dbs-n2 kernel: Kernel log daemon terminating.

Oct 6 22:42:13 dbs-n2 exiting on signal 15

Oct 6 22:45:52 dbs-n2 syslogd 1.4.1: restart.

Oct 6 22:45:52 dbs-n2 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Oct 6 22:45:52 dbs-n2 kernel: Bootdata ok (command line is ro root=LABEL=/ rhgb quiet)

Oct 6 22:45:52 dbs-n2 kernel: Linux version 2.6.18-164.el5xen (mockbuild@x86-003.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:59:52 EDT 2009

Oct 6 22:45:52 dbs-n2 kernel: BIOS-provided physical RAM map:

Oct 6 22:45:52 dbs-n2 kernel: Xen: 0000000000000000 - 00000001dc8fa000 (usable)

Oct 6 22:45:52 dbs-n2 kernel: DMI 2.7 present.

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x30] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x34] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x32] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x36] disabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled)

Oct 6 22:45:52 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x31] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x35] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)

Oct 6 22:45:53 dbs-n2 rpc.statd[4213]: Version 1.0.9 Starting

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x23] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x33] enabled)

Oct 6 22:45:53 dbs-n2 rpc.statd[4213]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x27] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x37] disabled)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

Oct 6 22:45:53 dbs-n2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])

Oct 6 22:45:53 dbs-n2 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23

Oct 6 22:45:53 dbs-n2 kernel: ACPI: IOAPIC (id[0x00] address[0xfec80000] gsi_base[24])

Oct 6 22:45:53 dbs-n2 kernel: IOAPIC[1]: apic_id 0, version 32, address 0xfec80000, GSI 24-47

Oct 6 22:45:53 dbs-n2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)

Oct 6 22:45:53 dbs-n2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

Oct 6 22:45:53 dbs-n2 kernel: Setting APIC routing to xen

Oct 6 22:45:53 dbs-n2 kernel: Using ACPI (MADT) for SMP configuration information

Oct 6 22:45:53 dbs-n2 kernel: Allocating PCI resources starting at e6000000 (gap: e4000000:1ac00000)

Oct 6 22:45:53 dbs-n2 kernel: Built 1 zonelists. Total pages: 1951994

Oct 6 22:45:53 dbs-n2 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet

Oct 6 22:45:53 dbs-n2 kernel: Initializing CPU#0

Oct 6 22:45:53 dbs-n2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)

Oct 6 22:45:53 dbs-n2 kernel: Xen reported: 2399.364 MHz processor.

Oct 6 22:45:53 dbs-n2 kernel: Console: colour VGA+ 80x25

Oct 6 22:45:54 dbs-n2 kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)

Oct 6 22:45:54 dbs-n2 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)

Oct 6 22:45:54 dbs-n2 kernel: Software IO TLB enabled:

Oct 6 22:45:54 dbs-n2 kernel: Aperture: 64 megabytes

Oct 6 22:45:54 dbs-n2 kernel: Kernel range: 0xffff88000a478000 - 0xffff88000e478000

Oct 6 22:45:54 dbs-n2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

Oct 6 22:45:54 dbs-n2 kernel: Memory: 7569128k/7807976k available (2490k kernel code, 229996k reserved, 1388k data, 184k init)

Oct 6 22:45:54 dbs-n2 hcid[4292]: Bluetooth HCI daemon

Oct 6 22:45:54 dbs-n2 kernel: Calibrating delay using timer specific routine.. 6001.33 BogoMIPS (lpj=12002673)

Oct 6 22:45:54 dbs-n2 kernel: Security Framework v1.0.0 initialized

Oct 6 22:45:54 dbs-n2 kernel: SELinux: Initializing.

Oct 6 22:45:54 dbs-n2 hcid[4292]: Register path:/org/bluez fallback:1

Oct 6 22:45:54 dbs-n2 sdpd[4296]: Bluetooth SDP daemon

Oct 6 22:45:54 dbs-n2 kernel: selinux_register_security: Registering secondary module capability

Oct 6 22:45:54 dbs-n2 kernel: Capability LSM initialized as secondary

Oct 6 22:45:54 dbs-n2 kernel: Mount-cache hash table entries: 256

Oct 6 22:45:54 dbs-n2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

Oct 6 22:45:54 dbs-n2 kernel: CPU: L2 cache: 256K

Oct 6 22:45:54 dbs-n2 kernel: CPU: L3 cache: 12288K

Oct 6 22:45:54 dbs-n2 kernel: CPU: Physical Processor ID: 0

Oct 6 22:45:54 dbs-n2 kernel: CPU: Processor Core ID: 0

Oct 6 22:45:54 dbs-n2 kernel: (SMP-)alternatives turned off

Oct 6 22:45:54 dbs-n2 kernel: ACPI: Core revision 20060707

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#1

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#2

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#3

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#4

Oct 6 22:45:54 dbs-n2 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#5

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#6

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#7

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#8

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#9

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#10

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#11

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#12

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#13

Oct 6 22:45:54 dbs-n2 kernel: Initializing CPU#14

Oct 6 22:45:55 dbs-n2 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.

Oct 6 22:45:55 dbs-n2 kernel: Brought up 16 CPUs

Oct 6 22:45:55 dbs-n2 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)

Oct 6 22:45:55 dbs-n2 kernel: Initializing CPU#15

Oct 6 22:45:55 dbs-n2 kernel: migration_cost=632

Oct 6 22:45:55 dbs-n2 kernel: checking if image is initramfs... it is

Oct 6 22:45:55 dbs-n2 kernel: Grant table initialized

Oct 6 22:45:55 dbs-n2 kernel: NET: Registered protocol family 16

Oct 6 22:45:55 dbs-n2 kernel: ACPI: bus type pci registered

Oct 6 22:45:55 dbs-n2 kernel: PCI: Not using MMCONFIG.

Oct 6 22:45:55 dbs-n2 kernel: PCI: Using configuration type 1

Oct 6 22:45:55 dbs-n2 kernel: ACPI: Interpreter enabled

Oct 6 22:45:55 dbs-n2 kernel: ACPI: Using IOAPIC for interrupt routing

Oct 6 22:45:55 dbs-n2 kernel: ACPI: No dock devices found.

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Root Bridge (0000:00)

Oct 6 22:45:55 dbs-n2 kernel: PCI: Transparent bridge - 0000:00:1e.0

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 *10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 10 *11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 *7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs *5 7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs *5 7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 10 11) *0, disabled.

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 7 *10 11)

Oct 6 22:45:55 dbs-n2 kernel: ACPI: PCI Interrupt Link (IRQs 5 *7 10 11)

Oct 6 22:45:55 dbs-n2 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay

Oct 6 22:45:55 dbs-n2 kernel: pnp: PnP ACPI init

Oct 6 22:45:55 dbs-n2 kernel: pnp: PnP ACPI: found 11 devices

Oct 6 22:45:55 dbs-n2 kernel: xen_mem: Initialising balloon driver.

Oct 6 22:45:55 dbs-n2 kernel: usbcore: registered new driver usbfs

Oct 6 22:45:55 dbs-n2 kernel: usbcore: registered new driver hub

Oct 6 22:45:55 dbs-n2 kernel: PCI: Using ACPI for IRQ routing

Oct 6 22:45:55 dbs-n2 kernel: PCI: If a device doesnt work, try "pci=routeirq". If it helps, post a report

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: Initializing

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: domain hash size = 128

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: protocols = UNLABELED CIPSOv4

Oct 6 22:45:55 dbs-n2 kernel: NetLabel: unlabeled traffic allowed by default

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved

Oct 6 22:45:55 dbs-n2 kernel: pnp: 00:01: ioport range 0x700-0x71f has been reserved

Oct 6 22:45:55 dbs-n2 kernel: PCI: Bridge: 0000:00:01.0

Oct 6 22:45:55 dbs-n2 kernel: IO window: 4000-4fff

Oct 6 22:45:55 dbs-n2 kernel: MEM window: fb700000-fbbfffff

Oct 6 22:45:55 dbs-n2 kernel: PREFETCH window 0x00000000e6000000-0x00000000e60fffff

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:02.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:03.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:04.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:05.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:06.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:07.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:08.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: 5000-5fff

Oct 6 22:45:56 dbs-n2 kernel: MEM window: fbf00000-fbffffff

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window 0x00000000e6100000-0x00000000e61fffff

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:09.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: PCI: Bridge: 0000:00:0a.0

Oct 6 22:45:56 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:56 dbs-n2 kernel: MEM window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.0

Oct 6 22:45:57 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f2000000-f5ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6200000-0x00000000e62fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.2

Oct 6 22:45:57 dbs-n2 kernel: IO window: disabled.

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f6000000-f9ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6300000-0x00000000e63fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1c.4

Oct 6 22:45:57 dbs-n2 kernel: IO window: 3000-3fff

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f1d00000-f1ffffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e6400000-0x00000000e64fffff

Oct 6 22:45:57 dbs-n2 kernel: PCI: Bridge: 0000:00:1e.0

Oct 6 22:45:57 dbs-n2 kernel: IO window: 2000-2fff

Oct 6 22:45:57 dbs-n2 kernel: MEM window: f1c00000-f1cfffff

Oct 6 22:45:57 dbs-n2 kernel: PREFETCH window 0x00000000e8000000-0x00000000efffffff

Oct 6 22:45:57 dbs-n2 hidd[4478]: Bluetooth HID daemon

Oct 6 22:45:57 dbs-n2 kernel: GSI 16 sharing vector 0x98 and IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.2 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.4 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:57 dbs-n2 kernel: NET: Registered protocol family 2

Oct 6 22:45:57 dbs-n2 kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

Oct 6 22:45:57 dbs-n2 kernel: TCP: Hash tables configured (established 262144 bind 65536)

Oct 6 22:45:57 dbs-n2 kernel: TCP reno registered

Oct 6 22:45:57 dbs-n2 kernel: audit: initializing netlink socket (disabled)

Oct 6 22:45:57 dbs-n2 kernel: type=2000 audit(1381070687.472:1): initialized

Oct 6 22:45:57 dbs-n2 kernel: VFS: Disk quotas dquot_6.5.1

Oct 6 22:45:57 dbs-n2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

Oct 6 22:45:57 dbs-n2 kernel: Initializing Cryptographic API

Oct 6 22:45:57 dbs-n2 kernel: alg: No test for crc32c (crc32c-generic)

Oct 6 22:45:57 dbs-n2 kernel: ksign: Installing public key data

Oct 6 22:45:57 dbs-n2 kernel: Loading keyring

Oct 6 22:45:57 dbs-n2 kernel: - Added public key 4BD2703D74512BB

Oct 6 22:45:57 dbs-n2 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)

Oct 6 22:45:57 dbs-n2 kernel: io scheduler noop registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler anticipatory registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler deadline registered

Oct 6 22:45:57 dbs-n2 kernel: io scheduler cfq registered (default)

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 last message repeated 9 times

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.2 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1c.4 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:45:58 dbs-n2 kernel: assign_interrupt_mode Found MSI capability

Oct 6 22:45:58 dbs-n2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU8 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU4 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU12 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU6 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU14 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU10 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU9 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU5 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU13 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU7 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU15 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: CPU11 (power states: C1[C1] C2[C3] C3[C3])

Oct 6 22:45:58 dbs-n2 kernel: ACPI: Thermal Zone (8 C)

Oct 6 22:45:58 dbs-n2 kernel: Real Time Clock Driver v1.12ac

Oct 6 22:45:58 dbs-n2 kernel: hpet_acpi_add: no address or irqs in _CRS

Oct 6 22:45:58 dbs-n2 kernel: Non-volatile memory driver v1.2

Oct 6 22:45:58 dbs-n2 kernel: Linux agpgart interface v0.101 (c) Dave Jones

Oct 6 22:45:58 dbs-n2 kernel: brd: module loaded

Oct 6 22:45:58 dbs-n2 kernel: Xen virtual console successfully installed as ttyS0

Oct 6 22:45:58 dbs-n2 kernel: Event-channel device installed.

Oct 6 22:45:58 dbs-n2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

Oct 6 22:45:58 dbs-n2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

Oct 6 22:45:58 dbs-n2 automount[4518]: lookup_read_master: lookup(nisplus): couldnt locate nis+ table auto.master

Oct 6 22:45:58 dbs-n2 kernel: ide-floppy driver 0.99.newide

Oct 6 22:45:58 dbs-n2 kernel: usbcore: registered new driver hiddev

Oct 6 22:45:58 dbs-n2 kernel: usbcore: registered new driver usbhid

Oct 6 22:45:58 dbs-n2 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver

Oct 6 22:45:58 dbs-n2 kernel: PNP: PS/2 Controller at 0x60,0x64 irq 1,12

Oct 6 22:45:59 dbs-n2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1

Oct 6 22:45:59 dbs-n2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12

Oct 6 22:45:59 dbs-n2 kernel: mice: PS/2 mouse device common for all mice

Oct 6 22:45:59 dbs-n2 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

Oct 6 22:45:59 dbs-n2 kernel: md: bitmap version 4.39

Oct 6 22:45:59 dbs-n2 kernel: TCP bic registered

Oct 6 22:45:59 dbs-n2 kernel: Initializing IPsec netlink socket

Oct 6 22:45:59 dbs-n2 kernel: NET: Registered protocol family 1

Oct 6 22:45:59 dbs-n2 kernel: NET: Registered protocol family 17

Oct 6 22:45:59 dbs-n2 kernel: Initalizing network drop monitor service

Oct 6 22:45:59 dbs-n2 kernel: Write protecting the kernel read-only data: 478k

Oct 6 22:45:59 dbs-n2 kernel: input: AT Translated Set 2 keyboard as /class/input/input0

Oct 6 22:45:59 dbs-n2 kernel: logips2pp: Detected unknown logitech mouse model 0

Oct 6 22:45:59 dbs-n2 kernel: GSI 17 sharing vector 0x49 and IRQ 17

Oct 6 22:45:59 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.7 -> GSI 20 (level, low) -> IRQ 17

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: debug port 1

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: irq 17, io mem 0xf1bf0000

Oct 6 22:45:59 dbs-n2 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004

Oct 6 22:45:59 dbs-n2 kernel: usb usb1: configuration #1 chosen from 1 choice

Oct 6 22:45:59 dbs-n2 kernel: hub 1-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 1-0:1.0: 8 ports detected

Oct 6 22:46:00 dbs-n2 kernel: USB Universal Host Controller Interface driver v3.0

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.0 -> GSI 20 (level, low) -> IRQ 17

Oct 6 22:46:00 dbs-n2 logd: : info: logd started with default configuration.

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2

Oct 6 22:46:00 dbs-n2 logd: : info: G_main_add_SignalHandler: Added signal handler for signal 15

Oct 6 22:46:00 dbs-n2 logd: : info: G_main_add_SignalHandler: Added signal handler for signal 15

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.0: irq 17, io base 0x00001000

Oct 6 22:46:00 dbs-n2 kernel: usb usb2: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 kernel: hub 2-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 2-0:1.0: 2 ports detected

Oct 6 22:46:00 dbs-n2 kernel: input: ImPS/2 Logitech Wheel Mouse as /class/input/input1

Oct 6 22:46:00 dbs-n2 kernel: GSI 18 sharing vector 0x51 and IRQ 18

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.1 -> GSI 23 (level, low) -> IRQ 18

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.1: irq 18, io base 0x00001020

Oct 6 22:46:00 dbs-n2 kernel: usb usb3: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 kernel: hub 3-0:1.0: USB hub found

Oct 6 22:46:00 dbs-n2 kernel: hub 3-0:1.0: 2 ports detected

Oct 6 22:46:00 dbs-n2 kernel: GSI 19 sharing vector 0x59 and IRQ 19

Oct 6 22:46:00 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.2 -> GSI 22 (level, low) -> IRQ 19

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4

Oct 6 22:46:00 dbs-n2 heartbeat: : info: Version 2 support: false

Oct 6 22:46:00 dbs-n2 kernel: uhci_hcd 0000:00:1d.2: irq 19, io base 0x00001040

Oct 6 22:46:00 dbs-n2 kernel: usb usb4: configuration #1 chosen from 1 choice

Oct 6 22:46:00 dbs-n2 heartbeat: : WARN: Logging daemon is disabled --enabling logging daemon is recommended

Oct 6 22:46:00 dbs-n2 kernel: hub 4-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 heartbeat: : info: **************************

Oct 6 22:46:01 dbs-n2 kernel: hub 4-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 heartbeat: : info: Configuration validated. Starting heartbeat 2.1.4

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:00:1d.3 -> GSI 23 (level, low) -> IRQ 18

Oct 6 22:46:01 dbs-n2 heartbeat: : info: heartbeat: version 2.1.4

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: UHCI Host Controller

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:00:1d.3: irq 18, io base 0x00001060

Oct 6 22:46:01 dbs-n2 kernel: usb usb5: configuration #1 chosen from 1 choice

Oct 6 22:46:01 dbs-n2 kernel: hub 5-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 kernel: hub 5-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 kernel: GSI 20 sharing vector 0x61 and IRQ 20

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:02:00.4 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: UHCI Host Controller

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: new USB bus registered, assigned bus number 6

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: port count misdetected? forcing to 2 ports

Oct 6 22:46:01 dbs-n2 kernel: uhci_hcd 0000:02:00.4: irq 20, io base 0x00003c00

Oct 6 22:46:01 dbs-n2 kernel: usb usb6: configuration #1 chosen from 1 choice

Oct 6 22:46:01 dbs-n2 kernel: hub 6-0:1.0: USB hub found

Oct 6 22:46:01 dbs-n2 kernel: hub 6-0:1.0: 2 ports detected

Oct 6 22:46:01 dbs-n2 kernel: SCSI subsystem initialized

Oct 6 22:46:01 dbs-n2 kernel: HP CISS Driver (v 3.6.20-RH3)

Oct 6 22:46:01 dbs-n2 kernel: GSI 21 sharing vector 0x69 and IRQ 21

Oct 6 22:46:01 dbs-n2 kernel: ACPI: PCI Interrupt 0000:05:00.0 -> GSI 28 (level, low) -> IRQ 21

Oct 6 22:46:01 dbs-n2 kernel: cciss0: <0x323a> at PCI 0000:05:00.0 IRQ 240 using DAC
Oct 6 22:46:01 dbs-n2 kernel: blocks= 286677120 block_size= 512
Oct 6 22:46:01 dbs-n2 kernel: heads= 255, sectors= 32, cylinders= 35132
Oct 6 22:46:01 dbs-n2 kernel:
Oct 6 22:46:01 dbs-n2 kernel: blocks= 286677120 block_size= 512
Oct 6 22:46:02 dbs-n2 kernel: heads= 255, sectors= 32, cylinders= 35132
Oct 6 22:46:02 dbs-n2 kernel:
Oct 6 22:46:02 dbs-n2 xinetd[4561]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Oct 6 22:46:02 dbs-n2 kernel: cciss/c0d0: p1 p2 p3 p4 < p5 >
Oct 6 22:46:02 dbs-n2 xinetd[4561]: Started working: 0 available services
Oct 6 22:46:02 dbs-n2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Oct 6 22:46:02 dbs-n2 kernel: QLogic Fibre Channel HBA Driver
Oct 6 22:46:02 dbs-n2 kernel: GSI 22 sharing vector 0x91 and IRQ 22
Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:0b:00.0 -> GSI 31 (level, low) -> IRQ 22

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Found an ISP2432, irq 22, iobase 0xffffc2000002a000

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Configuring PCI space...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Configure NVRAM parameters...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Verifying loaded RISC code...

Oct 6 22:46:02 dbs-n2 kernel: usb 6-1: new full speed USB device using uhci_hcd and address 2

Oct 6 22:46:02 dbs-n2 kernel: usb 6-1: configuration #1 chosen from 1 choice

Oct 6 22:46:02 dbs-n2 kernel: input: HP Virtual Keyboard as /class/input/input2

Oct 6 22:46:02 dbs-n2 kernel: input: USB HID v1.01 Keyboard on usb-0000:02:00.4-1

Oct 6 22:46:02 dbs-n2 kernel: input: HP Virtual Keyboard as /class/input/input3

Oct 6 22:46:02 dbs-n2 kernel: input: USB HID v1.01 Mouse on usb-0000:02:00.4-1

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Allocated (64 KB) for EFT...

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0: Allocated (1413 KB) for firmware dump...

Oct 6 22:46:02 dbs-n2 kernel: scsi0 : qla2xxx

Oct 6 22:46:02 dbs-n2 kernel: qla2xxx 0000:0b:00.0:

Oct 6 22:46:02 dbs-n2 kernel: QLogic Fibre Channel HBA Driver: 8.03.00.10.05.04-k

Oct 6 22:46:02 dbs-n2 kernel: QLogic HPAE311A - PCI-Express 4Gb Fibre Channel HBA

Oct 6 22:46:02 dbs-n2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0b:00.0 hdma+, host#=0, fw=4.04.09 (486)

Oct 6 22:46:02 dbs-n2 kernel: Initializing USB Mass Storage driver...

Oct 6 22:46:02 dbs-n2 kernel: usbcore: registered new driver usb-storage

Oct 6 22:46:02 dbs-n2 kernel: USB Mass Storage support registered.

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: uevent: version 1.0.3

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com

Oct 6 22:46:02 dbs-n2 kernel: device-mapper: dm-raid45: initialized v0.2594l

Oct 6 22:46:02 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:02 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:02 dbs-n2 kernel: SELinux: Disabled at runtime.

Oct 6 22:46:02 dbs-n2 kernel: type=1404 audit(1381070718.142:2): selinux=0 auid=4294967295 ses=4294967295

Oct 6 22:46:02 dbs-n2 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:03:00.0 -> GSI 16 (level, low) -> IRQ 16

Oct 6 22:46:02 dbs-n2 kernel: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 16, node addr 68b599787f22

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:03:00.1 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:02 dbs-n2 kernel: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 20, node addr 68b599787f24

Oct 6 22:46:02 dbs-n2 kernel: GSI 23 sharing vector 0x99 and IRQ 23

Oct 6 22:46:02 dbs-n2 kernel: ACPI: PCI Interrupt 0000:04:00.0 -> GSI 18 (level, low) -> IRQ 23

Oct 6 22:46:02 dbs-n2 kernel: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 23, node addr 68b599787f26

Oct 6 22:46:02 dbs-n2 kernel: GSI 24 sharing vector 0xA1 and IRQ 24

Oct 6 22:46:03 dbs-n2 kernel: ACPI: PCI Interrupt 0000:04:00.1 -> GSI 19 (level, low) -> IRQ 24

Oct 6 22:46:03 dbs-n2 kernel: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 24, node addr 68b599787f28

Oct 6 22:46:03 dbs-n2 kernel: input: PC Speaker as /class/input/input4

Oct 6 22:46:03 dbs-n2 kernel: ACPI: PCI Interrupt 0000:02:00.2 -> GSI 17 (level, low) -> IRQ 20

Oct 6 22:46:03 dbs-n2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

Oct 6 22:46:03 dbs-n2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

Oct 6 22:46:03 dbs-n2 kernel: Floppy drive(s): fd0 is 1.44M

Oct 6 22:46:03 dbs-n2 kernel: floppy0: no floppy controllers found

Oct 6 22:46:03 dbs-n2 kernel: Floppy drive(s): fd0 is 1.44M

Oct 6 22:46:03 dbs-n2 heartbeat: : info: Heartbeat generation: 1292541790

Oct 6 22:46:03 dbs-n2 kernel: floppy0: no floppy controllers found

Oct 6 22:46:03 dbs-n2 kernel: lp: driver loaded but no devices found

Oct 6 22:46:03 dbs-n2 kernel: NET: Registered protocol family 10

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1

Oct 6 22:46:03 dbs-n2 kernel: lo: Disabled Privacy Extensions

Oct 6 22:46:03 dbs-n2 kernel: IPv6 over IPv4 tunneling driver

Oct 6 22:46:03 dbs-n2 kernel: ACPI: Power Button (FF)

Oct 6 22:46:03 dbs-n2 kernel: md: Autodetecting RAID arrays.

Oct 6 22:46:03 dbs-n2 kernel: md: autorun ...

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1

Oct 6 22:46:03 dbs-n2 kernel: md: ... autorun DONE.

Oct 6 22:46:03 dbs-n2 heartbeat: : info: glib: ping heartbeat started.

Oct 6 22:46:03 dbs-n2 kernel: device-mapper: multipath: version 1.0.5 loaded

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p2, internal journal

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler

Oct 6 22:46:03 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:03 dbs-n2 heartbeat: : info: G_main_add_SignalHandler: Added signal handler for signal 17

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p5, internal journal

Oct 6 22:46:03 dbs-n2 heartbeat: : info: Local status now set to: 'up'

Oct 6 22:46:03 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:03 dbs-n2 kernel: kjournald starting. Commit interval 5 seconds

Oct 6 22:46:03 dbs-n2 kernel: EXT3 FS on cciss/c0d0p1, internal journal

Oct 6 22:46:03 dbs-n2 kernel: EXT3-fs: mounted filesystem with ordered data mode.

Oct 6 22:46:03 dbs-n2 kernel: Adding 16386292k swap on /dev/cciss/c0d0p3. Priority:-1 extents:1 across:16386292k

Oct 6 22:46:03 dbs-n2 kernel: IA-32 Microcode Update Driver: v1.14-xen

Oct 6 22:46:03 dbs-n2 kernel: Loading iSCSI transport class v2.0-871.

Oct 6 22:46:03 dbs-n2 kernel: 802.1Q VLAN Support v1.8 Ben Greear

Oct 6 22:46:03 dbs-n2 kernel: All bugs added by David S. Miller

Oct 6 22:46:03 dbs-n2 kernel: cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.

Oct 6 22:46:03 dbs-n2 kernel: iscsi: registered transport (cxgb3i)

Oct 6 22:46:04 dbs-n2 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.0.0 (March 21, 2009)

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth0

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth1

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth2

Oct 6 22:46:04 dbs-n2 kernel: cnic: Added CNIC device: eth3

Oct 6 22:46:04 dbs-n2 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.0.1e (June 22, 2009)

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (bnx2i)

Oct 6 22:46:04 dbs-n2 kernel: scsi1 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi2 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi3 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: scsi4 : Broadcom Offload iSCSI Initiator

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (tcp)

Oct 6 22:46:04 dbs-n2 kernel: iscsi: registered transport (iser)

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth0: using MSIX

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth1: using MSIX

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth1

Oct 6 22:46:04 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=eth1

Oct 6 22:46:04 dbs-n2 kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:46:04 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

Oct 6 22:46:04 dbs-n2 kernel: device-mapper: dm-log-clustered: (built Jul 27 2009 16:28:42) installed

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: Core ver 2.10

Oct 6 22:46:04 dbs-n2 kernel: NET: Registered protocol family 31

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HCI device and connection manager initialized

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HCI socket layer initialized

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: L2CAP ver 2.8

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Link dbs-n1:eth1 up.

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: L2CAP socket layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Status update for node dbs-n1: status up

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM socket layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 60 ms (> 50 ms) (GSource: 0x101b5858)

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM TTY layer initialized

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 up.

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: RFCOMM ver 1.8

Oct 6 22:46:04 dbs-n2 heartbeat: : info: Status update for node 192.191.0.167: status ping

Oct 6 22:46:04 dbs-n2 harc[4690]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:46:04 dbs-n2 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1

Oct 6 22:46:04 dbs-n2 heartbeat: : WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 120 ms (> 50 ms) (GSource: 0x101b8c18)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Link dbs-n2:eth1 up.

Oct 6 22:46:05 dbs-n2 heartbeat: : WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 110 ms (> 100 ms) before being called (GSource: 0x101b8458)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: G_SIG_dispatch: started at 1718072208 should have started at 1718072197

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Comm_now_up(): updating status to active

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Local status now set to: 'active'

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Starting child client "/usr/lib64/heartbeat/ipfail" (201,204)

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Status update for node dbs-n1: status active

Oct 6 22:46:05 dbs-n2 heartbeat: : info: Starting "/usr/lib64/heartbeat/ipfail" as uid 201 gid 204 (pid 4707)

Oct 6 22:46:05 dbs-n2 harc[4708]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:46:13 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: local resource transition completed.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: Initial resource acquisition complete (T_RESOURCES(us))

Oct 6 22:46:16 dbs-n2 heartbeat: : info: No local resources to acquire.

Oct 6 22:46:16 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:46:59 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:46:59 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:46:59 dbs-n2 heartbeat: : info: acquire local HA resources (standby).

Oct 6 22:46:59 dbs-n2 heartbeat: : info: local HA resource acquisition completed (standby).

Oct 6 22:46:59 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:47:00 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:49:08 dbs-n2 ipfail: : info: Telling other node that we have more visible ping nodes.

Oct 6 22:49:13 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:49:13 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:49:25 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:49:25 dbs-n2 heartbeat: : info: acquire all HA resources (standby).

Oct 6 22:49:25 dbs-n2 ResourceManager[4812]: info: Acquiring resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:49:25 dbs-n2 IPaddr[4839]: INFO: Resource is stopped

Oct 6 22:49:25 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 start

Oct 6 22:49:26 dbs-n2 IPaddr[4937]: INFO: Using calculated netmask for 192.191.0.22: 255.255.255.0

Oct 6 22:49:26 dbs-n2 IPaddr[4937]: INFO: eval ifconfig eth0:0 192.191.0.22 netmask 255.255.255.0 broadcast 192.191.0.255

Oct 6 22:49:26 dbs-n2 kernel: send_arp uses obsolete (PF_INET,SOCK_PACKET)

Oct 6 22:49:26 dbs-n2 IPaddr[4908]: INFO: Success

Oct 6 22:49:26 dbs-n2 Filesystem[5055]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:26 dbs-n2 Filesystem[5044]: INFO: Resource is stopped

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 start

Oct 6 22:49:26 dbs-n2 Filesystem[5130]: INFO: Running start for /dev/sda1 on /clusdsk

Oct 6 22:49:26 dbs-n2 Filesystem[5130]: ERROR: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:26 dbs-n2 Filesystem[5119]: ERROR: Illegal argument

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: ERROR: Return code 2 from /etc/ha.d/resource.d/Filesystem

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: CRIT: Giving up resources due to failure of Filesystem::/dev/sda1::/clusdsk::ext3

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:26 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:27 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:28 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:29 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:30 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:30 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:31 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:32 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:33 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:34 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:35 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:36 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Retrying failed stop operation

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: WARN: Machine reboot narrowly avoided!

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:49:37 dbs-n2 Filesystem[5601]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:49:37 dbs-n2 Filesystem[5601]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:49:37 dbs-n2 Filesystem[5590]: INFO: Success

Oct 6 22:49:37 dbs-n2 ResourceManager[4812]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:49:37 dbs-n2 IPaddr[5707]: INFO: ifconfig eth0:0 down

Oct 6 22:49:37 dbs-n2 IPaddr[5678]: INFO: Success

Oct 6 22:49:37 dbs-n2 heartbeat: : info: all HA resource acquisition completed (standby).

Oct 6 22:49:37 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:49:38 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:49:46 dbs-n2 gpm[5749]: *** info :

Oct 6 22:49:46 dbs-n2 gpm[5749]: Started gpm successfully. Entered daemon mode.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully dropped root privileges.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: avahi-daemon 0.6.16 starting up.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully called chroot().

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Successfully dropped remaining capabilities.

Oct 6 22:49:47 dbs-n2 kernel: Bridge firewalling registered

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Loading service file /services/sftp-ssh.service.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth1.IPv6 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::6ab5:99ff:fe78:7f24.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth1.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth1.IPv4 with address 10.9.0.21.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv6 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Network interface enumeration completed.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f24 on eth1.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for 10.9.0.21 on eth1.

Oct 6 22:49:47 dbs-n2 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.191.0.21 on eth0.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: Registering HINFO record with values 'X86_64'/'LINUX'.

Oct 6 22:49:47 dbs-n2 avahi-daemon[5871]: New relevant interface virbr0.IPv4 for mDNS.

Oct 6 22:49:47 dbs-n2 kernel: Netfilter messages via NETLINK v0.30.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.

Oct 6 22:49:48 dbs-n2 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.168.122.1 on virbr0.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: New relevant interface virbr0.IPv6 for mDNS.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0.

Oct 6 22:49:48 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::200:ff:fe00:0 on virbr0.

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: started, version 2.45 cachesize 150

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: compile time options: IPv6 GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: reading /etc/resolv.conf

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: using nameserver 192.191.0.20#53

Oct 6 22:49:49 dbs-n2 dnsmasq[6474]: read /etc/hosts - 6 addresses

Oct 6 22:49:49 dbs-n2 avahi-daemon[5871]: Server startup complete. Host name is dbs-n2.local. Local service cookie is 287356982.

Oct 6 22:49:50 dbs-n2 avahi-daemon[5871]: Service "SFTP File Transfer on dbs-n2" (/services/sftp-ssh.service) successfully established.

Oct 6 22:49:51 dbs-n2 xenstored: Checking store ...

Oct 6 22:49:52 dbs-n2 xenstored: Checking store complete.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Withdrawing address record for 192.191.0.21 on eth0.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: iface.c: interface_mdns_mcast_join() called but no local address available.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Interface eth0.IPv4 no longer relevant for mDNS.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Withdrawing address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: iface.c: interface_mdns_mcast_join() called but no local address available.

Oct 6 22:49:52 dbs-n2 avahi-daemon[5871]: Interface eth0.IPv6 no longer relevant for mDNS.

Oct 6 22:49:52 dbs-n2 kernel: device vif0.0 entered promiscuous mode

Oct 6 22:49:52 dbs-n2 kernel: xenbr0: topology change detected, propagating

Oct 6 22:49:52 dbs-n2 kernel: xenbr0: port 1(vif0.0) entering forwarding state

Oct 6 22:49:52 dbs-n2 kernel: bnx2: peth0: using MSIX

Oct 6 22:49:52 dbs-n2 kernel: ADDRCONF(NETDEV_UP): peth0: link is not ready

Oct 6 22:49:52 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=peth0

Oct 6 22:49:53 dbs-n2 heartbeat: : ERROR: glib: Error sending packet: Network is unreachable

Oct 6 22:49:53 dbs-n2 heartbeat: : info: glib: euid=99 egid=99

Oct 6 22:49:53 dbs-n2 heartbeat: : ERROR: write_child: write failure on ping 192.191.0.167.: Network is unreachable

Oct 6 22:49:54 dbs-n2 kernel: bnx2i: LOM is not enable to offload iSCSI connections, dev=peth0

Oct 6 22:49:55 dbs-n2 heartbeat: : ERROR: glib: Error sending packet: Network is unreachable

Oct 6 22:49:55 dbs-n2 heartbeat: : info: glib: euid=99 egid=99

Oct 6 22:49:55 dbs-n2 heartbeat: : ERROR: write_child: write failure on ping 192.191.0.167.: Network is unreachable

Oct 6 22:49:55 dbs-n2 kernel: bnx2: peth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

Oct 6 22:49:55 dbs-n2 kernel: ADDRCONF(NETDEV_CHANGE): peth0: link becomes ready

Oct 6 22:49:56 dbs-n2 kernel: device peth0 entered promiscuous mode

Oct 6 22:49:56 dbs-n2 kernel: xenbr0: topology change detected, propagating

Oct 6 22:49:56 dbs-n2 kernel: xenbr0: port 2(peth0) entering forwarding state

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv4 for mDNS.

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.191.0.21.

Oct 6 22:49:57 dbs-n2 avahi-daemon[5871]: Registering new address record for 192.191.0.21 on eth0.

Oct 6 22:49:57 dbs-n2 heartbeat: : WARN: node 192.191.0.167: is dead

Oct 6 22:49:57 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 dead.

Oct 6 22:49:57 dbs-n2 ipfail: : info: Status update: Node 192.191.0.167 now has status dead

Oct 6 22:49:57 dbs-n2 harc[6780]: info: Running /etc/ha.d/rc.d/status status

Oct 6 22:49:57 dbs-n2 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).

Oct 6 22:49:57 dbs-n2 kernel: Hangcheck: Using monotonic_clock().

Oct 6 22:49:58 dbs-n2 modclusterd: startup succeeded

Oct 6 22:49:58 dbs-n2 ipfail: : info: NS: We are dead. :<

Oct 6 22:49:58 dbs-n2 ipfail: : info: Link Status update: Link 192.191.0.167/192.191.0.167 now has status dead

Oct 6 22:49:58 dbs-n2 oddjobd: oddjobd startup succeeded

Oct 6 22:49:58 dbs-n2 saslauthd[6953]: detach_tty : master pid is: 6953

Oct 6 22:49:58 dbs-n2 saslauthd[6953]: ipc_init : listening on socket: /var/run/saslauthd/mux

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: New relevant interface eth0.IPv6 for mDNS.

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::6ab5:99ff:fe78:7f22.

Oct 6 22:49:58 dbs-n2 avahi-daemon[5871]: Registering new address record for fe80::6ab5:99ff:fe78:7f22 on eth0.

Oct 6 22:49:58 dbs-n2 ricci: startup succeeded

Oct 6 22:49:58 dbs-n2 smartd[6976]: smartd version 5.38 Copyright (C) 2002-8 Bruce Allen

Oct 6 22:49:58 dbs-n2 smartd[6976]: Home page is http://smartmontools.sourceforge.net/

Oct 6 22:49:58 dbs-n2 smartd[6976]: Opened configuration file /etc/smartd.conf

Oct 6 22:49:58 dbs-n2 smartd[6976]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices

Oct 6 22:49:58 dbs-n2 smartd[6976]: Problem creating device name scan list

Oct 6 22:49:58 dbs-n2 smartd[6976]: Problem creating device name scan list

Oct 6 22:49:58 dbs-n2 smartd[6976]: In the system's table of devices NO devices found to scan

Oct 6 22:49:58 dbs-n2 smartd[6976]: Monitoring 0 ATA and 0 SCSI devices

Oct 6 22:49:58 dbs-n2 smartd[6978]: smartd has fork()ed into background mode. New PID=6978.

Oct 6 22:49:59 dbs-n2 ipfail: : info: We are dead. :<

Oct 6 22:49:59 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:50:00 dbs-n2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found

Oct 6 22:50:00 dbs-n2 last message repeated 3 times

Oct 6 22:50:00 dbs-n2 heartbeat: : info: Link 192.191.0.167:192.191.0.167 up.

Oct 6 22:50:00 dbs-n2 heartbeat: : info: Status update for node 192.191.0.167: status ping

Oct 6 22:50:00 dbs-n2 ipfail: : info: Link Status update: Link 192.191.0.167/192.191.0.167 now has status up

Oct 6 22:50:00 dbs-n2 ipfail: : info: Status update: Node 192.191.0.167 now has status ping

Oct 6 22:50:00 dbs-n2 ipfail: : info: A ping node just came up.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Asking other side for ping node count.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Giving up because we were told that we have less ping nodes.

Oct 6 22:50:01 dbs-n2 ipfail: : info: Delayed giveup in 4 seconds.

Oct 6 22:50:04 dbs-n2 ipfail: : info: Aborted delayed giveup (3)

Oct 6 22:50:07 dbs-n2 hb_standby[7146]: Going standby .

Oct 6 22:50:08 dbs-n2 heartbeat: : info: dbs-n2 wants to go standby

Oct 6 22:50:08 dbs-n2 heartbeat: : info: standby: dbs-n1 can take our foreign resources

Oct 6 22:50:08 dbs-n2 heartbeat: : info: give up foreign HA resources (standby).

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:08 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:09 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:10 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:11 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:12 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:12 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:13 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:14 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:15 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:16 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:17 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:18 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Retrying failed stop operation

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: WARN: Machine reboot narrowly avoided!

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:50:19 dbs-n2 Filesystem[7590]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:50:19 dbs-n2 Filesystem[7590]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:50:19 dbs-n2 Filesystem[7579]: INFO: Success

Oct 6 22:50:19 dbs-n2 ResourceManager[7173]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:50:19 dbs-n2 IPaddr[7667]: INFO: Success

Oct 6 22:50:19 dbs-n2 heartbeat: : info: foreign HA resource release completed (standby).

Oct 6 22:50:19 dbs-n2 heartbeat: : info: Local standby process completed .

Oct 6 22:50:32 dbs-n2 heartbeat: : WARN: 1 lost packet(s) for

Oct 6 22:50:32 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:50:32 dbs-n2 heartbeat: : info: No pkts missing from dbs-n1!

Oct 6 22:50:32 dbs-n2 heartbeat: : info: Other node completed standby takeover of foreign resources.

Oct 6 22:50:45 dbs-n2 ipfail: : info: Ping node count is balanced.

Oct 6 22:50:45 dbs-n2 ipfail: : info: Giving up foreign resources (auto_failback).

Oct 6 22:50:45 dbs-n2 ipfail: : info: Delayed giveup in 4 seconds.

Oct 6 22:50:49 dbs-n2 ipfail: : info: giveup() called (timeout worked)

Oct 6 22:50:49 dbs-n2 heartbeat: : info: dbs-n2 wants to go standby

Oct 6 22:50:50 dbs-n2 heartbeat: : info: standby: dbs-n1 can take our foreign resources

Oct 6 22:50:50 dbs-n2 heartbeat: : info: give up foreign HA resources (standby).

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: info: Releasing resource group: dbs-n1 IPaddr::192.191.0.22/24/eth0 Filesystem::/dev/sda1::/clusdsk::ext3 oracledb

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:50 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:51 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:52 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:53 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:54 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:55 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:56 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:57 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:58 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:50:59 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Retrying failed stop operation

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/oracledb stop

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: ERROR: Return code 1 from /etc/ha.d/resource.d/oracledb

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: ERROR: Resource script for oracledb probably not LSB-compliant.

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: WARN: it (oracledb) MUST succeed on a stop when already stopped

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: WARN: Machine reboot narrowly avoided!

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/Filesystem /dev/sda1 /clusdsk ext3 stop

Oct 6 22:51:01 dbs-n2 Filesystem[8149]: WARNING: Couldn't find device . Expected /dev/??? to exist

Oct 6 22:51:01 dbs-n2 Filesystem[8149]: INFO: Running stop for /dev/sda1 on /clusdsk

Oct 6 22:51:01 dbs-n2 Filesystem[8138]: INFO: Success

Oct 6 22:51:01 dbs-n2 ResourceManager[7733]: info: Running /etc/ha.d/resource.d/IPaddr 192.191.0.22/24/eth0 stop

Oct 6 22:51:01 dbs-n2 IPaddr[8226]: INFO: Success

Oct 6 22:51:01 dbs-n2 heartbeat: : info: foreign HA resource release completed (standby).

Oct 6 22:51:01 dbs-n2 heartbeat: : info: Local standby process completed .

Oct 6 22:51:03 dbs-n2 heartbeat: : WARN: Standby in progress- new request from dbs-n1 ignored

Oct 6 22:51:14 dbs-n2 heartbeat: : WARN: 1 lost packet(s) for

Oct 6 22:51:14 dbs-n2 heartbeat: : info: remote resource transition completed.

Oct 6 22:51:14 dbs-n2 heartbeat: : info: No pkts missing from dbs-n1!

Oct 6 22:51:14 dbs-n2 heartbeat: : info: Other node completed standby takeover of foreign resources.

Oct 6 22:51:44 dbs-n2 heartbeat: : info: dbs-n1 wants to go standby

Oct 6 22:51:45 dbs-n2 heartbeat: : info: standby: acquire resources from dbs-n1

Oct 6 22:51:45 dbs-n2 heartbeat: : info: acquire local HA resources (standby).

Oct 6 22:51:45 dbs-n2 heartbeat: : info: local HA resource acquisition completed (standby).

Oct 6 22:51:45 dbs-n2 heartbeat: : info: Standby resource acquisition done .

Oct 6 22:51:45 dbs-n2 heartbeat: : info: remote resource transition completed.

#

大家能帮我分析问题出在哪里，及如何确认和排查吗？

wang.hy8166