PowerStore: Adding, removing or reseating drives might cause node reboot

Impacts all Non 500T models

This issue has been observed in PowerStoreOS 3.5 and higher.

Contact support if you need additional assistance.

The root cause of the panic is a software issue caused by the stock driver.

This is planned to be fixed in PowerStoreOS 5.x.

Troubleshooting

Drive OFFLINE prior  or close reboot:

LIVE: svc_journalctl -t xtremapp CODE_FUNC=cyc_platform_be_update_drive_info | grep "offline state change"
DC: ./cyc_triage.pl -j -- -t xtremapp CODE_FUNC=cyc_platform_be_update_drive_info | grep "offline state change"
Oct 16 00:04:18.419813 APPLID-A xtremapp[74468]: Oct 16 00:04:18.419782 D10 [log_id:0][13254(13396 nb_truck_10      0xf0030a67900 )][][INFO] x_cyc_disk.c:cyc_platform_be_update_drive_info:3001: drive DRV:0_0_8 offline state change sym_a=1 sym_b=1 local_a=0 local_b=0 drip=T, overall=OFFLINED
Oct 16 00:04:18.749937 APPLID-B xtremapp[59578]: Oct 16 00:04:18.742591 D11 [log_id:0][8242(8319 nb_truck_13      0xf0030a19e40 )][][INFO] x_cyc_disk.c:cyc_platform_be_update_drive_info:3001: drive DRV:0_0_8 offline state change sym_a=1 sym_b=1 local_a=0 local_b=0 drip=T, overall=OFFLINED

panic_log reporting pcieport bouncing (not present / present)

LIVE: svc_journalclt -t panic_log | grep "pcieport .*: pciehp: Slot.*: Card"
DC: ./cyc_triage.pl -j -- -t panic_log | grep "pcieport .*: pciehp: Slot.*: Card"
Oct 16 00:36:37.497402 APPLID-B panic_log[3530]: [18509940.864666] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.503942 APPLID-B panic_log[3530]: [18509942.266292] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.510129 APPLID-B panic_log[3530]: [18510184.427646] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.510234 APPLID-B panic_log[3530]: [18510200.919167] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.516163 APPLID-B panic_log[3530]: [18510440.552705] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.516252 APPLID-B panic_log[3530]: [18510457.530572] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.516667 APPLID-B panic_log[3530]: [18510696.712493] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.516738 APPLID-B panic_log[3530]: [18510713.831945] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.522359 APPLID-B panic_log[3530]: [18510971.937449] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.522426 APPLID-B panic_log[3530]: [18510982.976310] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.522864 APPLID-B panic_log[3530]: [18511241.860098] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.528042 APPLID-B panic_log[3530]: [18511253.052530] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 00:36:37.528501 APPLID-B panic_log[3530]: [18511512.003406] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 00:36:37.528570 APPLID-B panic_log[3530]: [18511523.323850] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 17:48:49.872432 APPLID-B panic_log[3521]: [61589.604211] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 17:48:49.872504 APPLID-B panic_log[3521]: [61606.019689] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 17:52:33.511068 APPLID-A panic_log[3546]: [  438.935050] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 17:52:33.511134 APPLID-A panic_log[3546]: [  450.345004] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 17:52:33.517359 APPLID-A panic_log[3546]: [  694.414935] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 17:52:33.522427 APPLID-A panic_log[3546]: [  705.914512] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 17:57:46.443660 APPLID-A panic_log[3318]: [  438.935050] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 17:57:46.443725 APPLID-A panic_log[3318]: [  450.345004] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 17:57:46.450021 APPLID-A panic_log[3318]: [  694.414935] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 17:57:46.450083 APPLID-A panic_log[3318]: [  705.914512] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 18:06:31.747263 APPLID-B panic_log[3522]: [  450.948824] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 18:06:31.747323 APPLID-B panic_log[3522]: [  461.863488] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present
Oct 16 18:06:31.777154 APPLID-B panic_log[3522]: [  721.655012] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card not present
Oct 16 18:06:31.777246 APPLID-B panic_log[3522]: [  732.848249] pcieport 10000:02:08.0: pciehp: Slot(8-1): Card present

Find the corresponding slot / drive:

LIVE: svc_diag list --nvme_drive
DC: node_[ab]/command_output/nvme_drive.py_--list.txt

Example:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 8 a | PWRON | UP         L0         | 10001:02:08.0 10000:0b:00.0 | Samsung PM1723b      -                    -                | -        | -           -             | -       
   b | PWRON | UP         L0         | 10001:02:08.0 10000:0b:00.0 | Samsung PM1723b      -                    -                | -        | -           -             | -       
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------