Device-mapper multipath failing path keeps happen

Hello,

Recently find the MD3220 multipath keeps failing and caused the performance drop.  However that is no idea what happen about it.  Hope someone from here can help us. 

OS is RHEL 6.5 now

Following is some config at log which got for you have some idea.

multipath.conf

blacklist {
#       devnode ".*"
        device {
                vendor  ".*"
                product ".*"
        }
}
blacklist_exceptions {
        device {
                vendor  "DELL"
                product "MD32xx.*"
        }
}

# multipath -ll
3690b11c00029d7c30000094e52f3b80c dm-5 DELL,MD32xx
size=2.7T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=4 status=active
| `- 2:0:6:13 sdf 8:80  active ghost running
`-+- policy='round-robin 0' prio=3 status=enabled
  `- 2:0:5:13 sdj 8:144 active ready running
36842b2b0006bcc80000009e152f2ae36 dm-7 DELL,MD32xx
size=2.7T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| `- 2:0:6:12 sde 8:64  active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 2:0:5:12 sdi 8:128 active ghost running
36842b2b0006bcc80000009ee52f2d9e1 dm-4 DELL,MD32xx
size=2.2T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=4 status=active
| `- 2:0:6:14 sdg 8:96  active ghost running
`-+- policy='round-robin 0' prio=3 status=enabled
  `- 2:0:5:14 sdk 8:160 active ready running
3690b11c00029d7c30000094a52f38c61 dm-6 DELL,MD32xx
size=2.7T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| `- 2:0:6:11 sdd 8:48  active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 2:0:5:11 sdh 8:112 active ghost running

# dmsetup -v ls
onapp--k7gi50w260rtoq-joc6y7i8u0o0n2    (253:35)
onapp--gydjut6u1btno1-l2g5lur4liby5r    (253:30)
lvg1-lv3        (253:2)
onapp--k7gi50w260rtoq-toj8fjp02o5af6    (253:43)
onapp--k7gi50w260rtoq-b1lkajwwwo35po    (253:39)
lvg1-lv2        (253:3)
onapp--gydjut6u1btno1-ct28f9yzhqlj5l    (253:58)
onapp--k7gi50w260rtoq-sme5a63a290n3y    (253:44)
onapp--k7gi50w260rtoq-xirt3kd8o9bkcd    (253:29)
lvg1-swap       (253:1)
lvg1-lv1        (253:0)
onapp--pn20akbx718ln1-jnup8v1o086t3q    (253:24)
onapp--gydjut6u1btno1-e09k44becib6ke    (253:73)
onapp--k7gi50w260rtoq-backup--mcbinzumraztez-cow        (253:15)
onapp--k7gi50w260rtoq-nybttu2as6vuhr    (253:28)
36842b2b0006bcc80000009e152f2ae36       (253:7)
onapp--gydjut6u1btno1-eggsdgfjoux9lu    (253:32)
onapp--gydjut6u1btno1-guloxnxncmc1ee    (253:21)
36842b2b0006bcc80000009ee52f2d9e1       (253:4)
onapp--gydjut6u1btno1-qoekmyoiqfd6at    (253:20)
onapp--k7gi50w260rtoq-backup--mcbinzumraztez    (253:8)
onapp--k7gi50w260rtoq-ft6r2vitww5zmh-real       (253:9)
3690b11c00029d7c30000094a52f38c61       (253:6)

--------------------------------------------

You will see keeps I/O error, dev sdf, with path 8:80 and with switch to 8:144: reinstated and then jump back to 8:80.  

Feb 14 14:21:22 n03 kernel: end_request: I/O error, dev sdf, sector 1222224624
Feb 14 14:21:22 n03 kernel: device-mapper: multipath: Failing path 8:80.
Feb 14 14:21:22 n03 multipathd: 8:80: mark as failed
Feb 14 14:21:22 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: remaining active paths: 1
Feb 14 14:21:23 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdj - rdac checker reports path is up
Feb 14 14:21:23 n03 multipathd: 8:144: reinstated
Feb 14 14:21:23 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: load table [0 5855354880 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 2 2 round-robin 0 1 1 8:80 1 round-robin 0 1 1 8:144 1]
Feb 14 14:21:23 n03 kernel: device-mapper: multipath: Failing path 8:80.
Feb 14 14:21:26 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdf - rdac checker reports path is ghost
Feb 14 14:21:26 n03 multipathd: 8:80: reinstated
Feb 14 14:21:26 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: remaining active paths: 2
Feb 14 14:21:26 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: switch to path group #1
Feb 14 14:21:35 n03 kernel: sd 2:0:6:13: rdac: array , ctlr 1, queueing MODE_SELECT command
Feb 14 14:21:37 n03 kernel: sd 2:0:6:13: rdac: array , ctlr 1, MODE_SELECT completed
Feb 14 14:21:38 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdj - rdac checker reports path is ghost
Feb 14 14:21:38 n03 multipathd: 8:144: reinstated
Feb 14 14:21:38 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: load table [0 5855354880 multipath 3 pg_init_retries 50 queue_if_no_path1 rdac 2 1 round-robin 0 1 1 8:80 1 round-robin 0 1 1 8:144 1]
Feb 14 14:21:41 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdf - rdac checker reports path is up
Feb 14 14:21:41 n03 multipathd: 8:80: reinstated
Feb 14 14:21:43 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdj - rdac checker reports path is up
Feb 14 14:21:43 n03 multipathd: 8:144: reinstated
Feb 14 14:21:43 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: load table [0 5855354880 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 2 1 round-robin 0 1 1 8:80 1 round-robin 0 1 1 8:144 1]
Feb 14 14:21:43 n03 kernel: sd 2:0:6:13: rdac: array , ctlr 1, queueing MODE_SELECT command
Feb 14 14:21:46 n03 kernel: sd 2:0:6:13: rdac: array , ctlr 1, MODE_SELECT completed
Feb 14 14:21:48 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdj - rdac checker reports path is ghost
Feb 14 14:21:48 n03 multipathd: 8:144: reinstated
Feb 14 14:21:48 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: load table [0 5855354880 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 2 1 round-robin 0 1 1 8:80 1 round-robin 0 1 1 8:144 1]
Feb 14 14:21:53 n03 multipathd: 3690b11c00029d7c30000094e52f3b80c: sdj - rdac checker reports path is up
Feb 14 14:21:53 n03 multipathd: 8:144: reinstated

-------------------------------------

Hope someone can have idea for me to trace the problem.

Thanks!