PowerPath use-after-free 패닉

VPLEX NDU와 VPLEX 디렉터 충돌

OS: Red Hat Enterprise Linux 6.4
소프트웨어: PowerPath for Linux 6.0

VPLEX NDU 중 Red Hat Enterprise Linux 서버 패닉 발생

Pid: 632, comm: fc_wq_2 Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL380 Gen9
RIP: 0010:[<ffffffffa0164c83>]  [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
RSP: 0018:ffff88204d60bcc0  EFLAGS: 00010286
RAX: 006e647300000010 RBX: ffff8804d485e938 RCX: ffffffff81b0b790
RDX: ffffffff81ebc5dc RSI: ffffffff81388a90 RDI: ffff8804d485e938
RBP: ffff88204d60bce0 R08: 0000000000000000 R09: ffffffff816490c0
R10: 0000000000000000 R11: 0000000000000000 R12: ffff881a02cb8000
R13: ffff881a02cb8010 R14: ffff88204f282060 R15: ffff88204da53010
FS:  0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fa01228dd50 CR3: 000000031a926000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fc_wq_2 (pid: 632, threadinfo ffff88204d60a000, task ffff88204f0d8040)
Stack:
 ffff88204f282060 ffff8804d485e938 ffffffffa016a9c8 ffffffff81b0bca0
<d> ffff88204d60bd00 ffffffff8136c72f ffff8804d485e998 ffff8804d485e938
<d> ffff88204d60bd20 ffffffff8136c89d ffff8804d485e938 ffff880c20c39c28
Call Trace:
 [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
 [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
 [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
 [<ffffffff813692ad>] device_del+0x12d/0x1e0
 [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
 [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
 [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
 [<ffffffffa005e570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
 [<ffffffffa005e596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
 [<ffffffff81098100>] worker_thread+0x170/0x2a0
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e78e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 00 48 89 fb 48 c7 c7 80 b7 b0 81 e8 a8 28 f4 e0 48 89 df e8 b0 76 20 e1 49 89 c4 48 8b 40 08 48 c7 c6 90 8a 38 81 4d 8d 6c 24 10 <48> 8b 78 08 e8 34 13 11 e1 49 8b 44 24 08 31 f6 48 8b 78 08 e8
RIP  [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
 RSP <ffff88204d60bcc0>

PowerPath 버그

VPLEX NDU 중에 VPLEX 디렉터가 돌아온 후 몇 초 동안 응답하지 않습니다. 이로 인해 디바이스 열기와 관련된 PowerPath의 버그가 노출되었으며, 이로 인해 PowerPath에서 추가 릴리스가 발행되었습니다. 이 추가 릴리스는 use-after-free 상태와 Linux 커널 패닉을 초래합니다.

PowerPath 6.1 HF1 및 PowerPath의 향후 릴리스에는 이 버그에 대한 수정 사항이 있습니다.

Red Hat과 PowerPath 엔지니어링 팀 모두 VPLEX NDU와 독립적으로 랩에서 조건을 재현할 수 있습니다. 고객 역추적 외에도 몇 가지 다른 패닉 역추적이 관찰되었습니다.
use-after-free의 특성상 실제 패닉은 여러 곳에서 볼 수 있습니다.

고객 역추적 #2

Pid: 0, comm: swapper Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL380 Gen9
RIP: 0010:[<ffffffff8137f2fc>]  [<ffffffff8137f2fc>] scsi_finish_command+0xac/0x130
RSP: 0018:ffff880028203e40  EFLAGS: 00010286
RAX: 0000000000010001 RBX: ffff88106dd7b580 RCX: 000000000000721b
RDX: ffff88402005c800 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff880028203e60 R08: ffff884021a8d288 R09: ffff8840269fd090
R10: ffff880028203d08 R11: 0000000000000001 R12: 0000000000004000
R13: ffff88401f46d800 R14: ffff8840219d2000 R15: 0000000000000004
FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000010079 CR3: 0000000ecc6a0000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a8d020)
Stack:
 ffff88106dd7b580 0000000000002002 0000000000007530 0000000000000005
<d> ffff880028203e90 ffffffff81389637 ffff880028203ea0 ffffffff81a830a0
<d> 0000000000000020 0000000000000100 ffff880028203ec0 ffffffff812787f5
Call Trace:
 <IRQ>
 [<ffffffff81389637>] scsi_softirq_done+0x147/0x170
 [<ffffffff812787f5>] blk_done_softirq+0x85/0xa0
 [<ffffffff8107d901>] __do_softirq+0xc1/0x1e0
 [<ffffffff810eac70>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8100c38c>] call_softirq+0x1c/0x30
 [<ffffffff8100fbd5>] do_softirq+0x65/0xa0
 [<ffffffff8107d7b5>] irq_exit+0x85/0x90
 [<ffffffff81533ba5>] do_IRQ+0x75/0xf0
 [<ffffffff8100ba53>] ret_from_intr+0x0/0x11
 <EOI>
 [<ffffffff812eab5e>] ? intel_idle+0xde/0x170
 [<ffffffff812eab41>] ? intel_idle+0xc1/0x170
 [<ffffffff81426167>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009fc6>] cpu_idle+0xb6/0x110
 [<ffffffff8151067a>] rest_init+0x7a/0x80
 [<ffffffff81c29f8f>] start_kernel+0x424/0x430
 [<ffffffff81c2933a>] x86_64_start_reservations+0x125/0x129
 [<ffffffff81c29453>] x86_64_start_kernel+0x115/0x124
Code: 77 7c 48 8b 83 80 00 00 00 44 8b 63 68 83 78 44 02 74 2e 48 8b 90 b0 00 00 00 31 c0 48 85 d2 74 0a 48 8b 82 c8 02 00 00 48 8b 00 <48> 8b 40 78 48 85 c0 74 33 48 89 df ff d0 44 39 e0 74 29 41 89
RIP  [<ffffffff8137f2fc>] scsi_finish_command+0xac/0x130
 RSP <ffff880028203e40>
CR2: 0000000000010079

EMC 재현 역추적 #2

Pid: 289, comm: fc_wq_1 Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 Dell Inc. PowerEdge 2950/0H603H
RIP: 0010:[<ffffffff8120b2ec>]  [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
RSP: 0018:ffff880225c41c10  EFLAGS: 00010286
RAX: ffff880225c40000 RBX: 0000000000000000 RCX: ffff88000cc501d8
RDX: 0000000000000000 RSI: ffffffff817e9361 RDI: 0000000000000000
RBP: ffff880225c41c20 R08: 0000000000001033 R09: ffffc90012e68000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817e9361
R13: 0000000000000000 R14: ffff880226dfa060 R15: ffff880225c15010
FS:  0000000000000000(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000068 CR3: 0000000229e15000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fc_wq_1 (pid: 289, threadinfo ffff880225c40000, task ffff880226dcd520)
Stack:
 0000000000000000 ffffffff817e9361 ffff880225c41c40 ffffffff8120c3d3
<d> ffffffff81b09ce0 ffff8802250fe420 ffff880225c41c70 ffffffff8120dd00
<d> ffff8802250fe410 ffff8802250fe410 ffff88012185c138 ffff8802250fe410
Call Trace:
 [<ffffffff8120c3d3>] sysfs_get_dirent+0x33/0x80
 [<ffffffff8120dd00>] sysfs_remove_group+0x30/0x100
 [<ffffffff81371559>] dpm_sysfs_remove+0x19/0x20
 [<ffffffff813691c9>] device_del+0x49/0x1e0
 [<ffffffffa0276ca4>] sd_remove+0x64/0xc0 [sd_mod]
 [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
 [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
 [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
 [<ffffffff813692ad>] device_del+0x12d/0x1e0
 [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
 [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
 [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
 [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
 [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
 [<ffffffff81098100>] worker_thread+0x170/0x2a0
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e78e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: c6 48 c7 c7 c0 08 7d 81 31 c0 e8 14 e1 31 00 0f 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 54 53 0f 1f 44 00 00 <48> 8b 47 68 49 89 f4 eb 1e 0f 1f 00 48 8b 70 e0 4c 89 e7 48 8d
RIP  [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
 RSP <ffff880225c41c10>
CR2: 0000000000000068

EMC 재현 역추적 #4

Pid: 290, comm: fc_wq_1 Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 Dell Inc. PowerEdge 2950/0H603H
RIP: 0010:[<ffffffff8137eb79>]  [<ffffffff8137eb79>] scsi_device_get+0x19/0xe0
RSP: 0018:ffff880226c93aa0  EFLAGS: 00010286
RAX: ffff880226c92000 RBX: ffff880225b45400 RCX: 000000000000001e
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000a300000001
RBP: ffff880226c93ac0 R08: fffffffcfffffffc R09: ffffffb0ffffffb0
R10: fffffffcfffffffc R11: fffffffcfffffffc R12: ffff88022762e958
R13: 0000000000000002 R14: ffff88022762e808 R15: 00000000fffffffa
FS:  0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000a3000005a1 CR3: 00000002097bb000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fc_wq_1 (pid: 290, threadinfo ffff880226c92000, task ffff880227443520)
Stack:
 ffff8800138c7210 ffff880225b45400 ffff88022762e958 0000000000000002
<d> ffff880226c93ae0 ffffffffa0276449 ffff880223dab400 ffff880223dab400
<d> ffff880226c93b20 ffffffffa02777a8 ffff880226c93b00 ffff88022762ea30
Call Trace:
 [<ffffffffa0276449>] __scsi_disk_get+0x39/0x50 [sd_mod]
 [<ffffffffa02777a8>] sd_open+0x48/0x1d0 [sd_mod]
 [<ffffffffa079fa96>] emcp_sd_bdops_open+0x26/0x60 [emcp]
 [<ffffffffa07bb9cc>] emcp_remove+0x3ec/0x8a0 [emcp]
 [<ffffffffa07bbef5>] emcp_chg_device_notify+0x75/0x90 [emcp]
 [<ffffffff815300d5>] notifier_call_chain+0x55/0x80
 [<ffffffff810a529a>] __blocking_notifier_call_chain+0x5a/0x80
 [<ffffffff810a52d6>] blocking_notifier_call_chain+0x16/0x20
 [<ffffffff8136c718>] __device_release_driver+0x58/0xe0
 [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
 [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
 [<ffffffff813692ad>] device_del+0x12d/0x1e0
 [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
 [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
 [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
 [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
 [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
 [<ffffffff81098100>] worker_thread+0x170/0x2a0
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e78e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: c3 90 48 89 df e8 b8 e0 d3 ff eb dd 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f 1f 44 00 00 <83> bf a0 05 00 00 04 48 89 fb 75 1b b8 fa ff ff ff 48 8b 5d e8
RIP  [<ffffffff8137eb79>] scsi_device_get+0x19/0xe0
 RSP <ffff880226c93aa0>
CR2: 000000a3000005a1

EMC 재현 역추적 #5

[49952.667188] Pid: 289, comm: fc_wq_1 Tainted: P           ---------------    2.6.32-504.23.4.el6.x86_64 #1 Dell Inc. PowerEdge 2950/0H603H
[49952.667188] RIP: 0010:[<ffffffff81293be9>]  [<ffffffff81293be9>] strnlen+0x9/0x40
[49952.667188] RSP: 0018:ffff88022630d930  EFLAGS: 00010086
[49952.667188] RAX: ffffffff817bcc11 RBX: ffffffff81eb9c00 RCX: 0000000000000002
[49952.667188] RDX: fffffffcfffffffc RSI: ffffffffffffffff RDI: fffffffcfffffffc
[49952.667188] RBP: ffff88022630d930 R08: 0000000000000073 R09: 00000000fffffff8
[49952.667188] R10: 0000000000000006 R11: 00000000000000c8 R12: ffffffff81eb9857
[49952.667188] R13: fffffffcfffffffc R14: 00000000ffffffff R15: 0000000000000000
[49952.667188] FS:  0000000000000000(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
[49952.667188] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[49952.667188] CR2: fffffffcfffffffc CR3: 000000022a50e000 CR4: 00000000000007e0
[49952.667188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[49952.667188] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[49952.667188] Process fc_wq_1 (pid: 289, threadinfo ffff88022630c000, task ffff8802276c9520)
[49952.667188] Stack:
[49952.667188]  ffff88022630d970 ffffffff81294ed0 0000000000000000 ffffffff81eb9857
[49952.667188] <d> ffffffffa084f498 ffffffffa084f496 ffff88022630dae0 ffffffff81eb9c00
[49952.667188] <d> ffff88022630da10 ffffffff81296908 0000000000000004 0000000affffffff
[49952.667188] Call Trace:
[49952.667188]  [<ffffffff81294ed0>] string+0x40/0x100
[49952.667188]  [<ffffffff81296908>] vsnprintf+0x218/0x5e0
[49952.667188]  [<ffffffff812a249e>] ? percpu_counter_destroy+0x3e/0x50
[49952.667188]  [<ffffffff81296d91>] vscnprintf+0x11/0x30
[49952.667188]  [<ffffffff81075c16>] vprintk+0xc6/0x560
[49952.667188]  [<ffffffff8128e3bd>] ? kobject_release+0x8d/0x240
[49952.667188]  [<ffffffff8128e330>] ? kobject_release+0x0/0x240
[49952.667188]  [<ffffffff8128f8a7>] ? kref_put+0x37/0x70
[49952.667188]  [<ffffffff81529425>] printk+0x41/0x44
[49952.667188]  [<ffffffffa0838d41>] ? PowerGetSemaNew+0x51/0x210 [emcp]
[49952.667188]  [<ffffffffa084b24d>] emcp_remove+0x78d/0x8d0 [emcp]
[49952.667188]  [<ffffffffa084b405>] emcp_chg_device_notify+0x75/0x90 [emcp]
[49952.667188]  [<ffffffff815300d5>] notifier_call_chain+0x55/0x80
[49952.667188]  [<ffffffff810a529a>] __blocking_notifier_call_chain+0x5a/0x80
[49952.667188]  [<ffffffff810a52d6>] blocking_notifier_call_chain+0x16/0x20
[49952.667188]  [<ffffffff8136c718>] __device_release_driver+0x58/0xe0
[49952.667188]  [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
[49952.667188]  [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
[49952.667188]  [<ffffffff813692ad>] device_del+0x12d/0x1e0
[49953.026077]  rport-2:0-4: blocked FC remote port time out: removing rport
[49952.667188]  [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
[49952.667188]  [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
[49952.667188]  [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
[49952.667188]  [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[49952.667188]  [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[49952.667188]  [<ffffffff81098100>] worker_thread+0x170/0x2a0
[49952.667188]  [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[49952.667188]  [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[49952.667188]  [<ffffffff8109e78e>] kthread+0x9e/0xc0
[49952.667188]  [<ffffffff8100c28a>] child_rip+0xa/0x20
[49952.667188]  [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[49952.667188]  [<ffffffff8100c280>] ? child_rip+0x0/0x20
[49952.667188] Code: 66 90 48 83 c2 01 80 3a 00 75 f7 48 89 d0 48 29 f8 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 f6 48 89 e5 74 2e <80> 3f 00 74 29 48 83 ee 01 48 89 f8 eb 12 66 0f 1f 84 00 00 00
[49952.667188] RIP  [<ffffffff81293be9>] strnlen+0x9/0x40
[49952.667188]  RSP <ffff88022630d930>
[49952.667188] CR2: fffffffcfffffffc

EMC 재현 역추적 #6

[ 1264.376313] Pid: 289, comm: fc_wq_1 Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 Dell Inc. PowerEdge 2950/0H603H
[ 1264.376313] RIP: 0010:[<ffffffff81372ae2>]  [<ffffffff81372ae2>] device_pm_remove+0x42/0xb0
[ 1264.376313] RSP: 0018:ffff88022635bc70  EFLAGS: 00010286
[ 1264.376313] RAX: ffff88022419c0b0 RBX: ffff88022419c010 RCX: 0000000000000000
[ 1264.376313] RDX: ffff880223e39cc8 RSI: 0000000000000000 RDI: ffffffff81b09d80
[ 1264.376313] RBP: ffff88022635bc80 R08: 000000000009d39c R09: 0000000000000000
[ 1264.376313] R10: 0000000000000000 R11: 00000000000000be R12: ffff88022409fee0
[ 1264.376313] R13: ffff88022419c010 R14: ffff880227762860 R15: ffff880226336010
[ 1264.376313] FS:  0000000000000000(0000) GS:ffff880028340000(0000) knlGS:0000000000000000
[ 1264.376313] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 1264.376313] CR2: 0000000000000008 CR3: 0000000001a85000 CR4: 00000000000007e0
[ 1264.376313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1264.376313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1264.376313] Process fc_wq_1 (pid: 289, threadinfo ffff88022635a000, task ffff880226359520)
[ 1264.376313] Stack:
[ 1264.376313]  ffff8802091fc138 ffff88022419c010 ffff88022635bcb0 ffffffff813691c1
[ 1264.376313] <d> ffffffff81b0bca0 ffff8802091fc138 ffff88022419c000 ffff88022419c010
[ 1264.376313] <d> ffff88022635bce0 ffffffffa0276ca4 ffff880227762860 ffff8802091fc138
[ 1264.376313] Call Trace:
[ 1264.376313]  [<ffffffff813691c1>] device_del+0x41/0x1e0
[ 1264.376313]  [<ffffffffa0276ca4>] sd_remove+0x64/0xc0 [sd_mod]
[ 1264.376313]  [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
[ 1264.376313]  [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
[ 1264.376313]  [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
[ 1264.376313]  [<ffffffff813692ad>] device_del+0x12d/0x1e0
[ 1264.376313]  [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
[ 1264.376313]  [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
[ 1264.376313]  [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
[ 1264.376313]  [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[ 1264.376313]  [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[ 1264.376313]  [<ffffffff81098100>] worker_thread+0x170/0x2a0
[ 1264.376313]  [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[ 1264.376313]  [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[ 1264.693138]  rport-2:0-12: blocked FC remote port time out: removing rport
[ 1264.376313]  [<ffffffff8109e78e>] kthread+0x9e/0xc0
[ 1264.376313]  [<ffffffff8100c28a>] child_rip+0xa/0x20
[ 1264.376313]  [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[ 1264.376313]  [<ffffffff8100c280>] ? child_rip+0x0/0x20
[ 1264.376313] Code: fb 75 51 48 c7 c7 80 9d b0 81 e8 da 83 1b 00 48 8b 8b a0 00 00 00 48 8b 93 a8 00 00 00 48 8d 83 a0 00 00 00 48 c7 c7 80 9d b0 81 <48> 89 51 08 48 89 0a 48 89 83 a0 00 00 00 48 89 83 a8 00 00 00
[ 1264.376313] RIP  [<ffffffff81372ae2>] device_pm_remove+0x42/0xb0
[ 1264.376313]  RSP <ffff88022635bc70>
[ 1264.376313] CR2: 0000000000000008