DELL EMC VNX: Storage Pool user capacity changed after SP reboot (User Correctable)

After a SP reboot or panic, the storagepool user capacity was changed.

To verify the issue, you may need two sets of SPcollects before and after the reboot, as well as the ktrace log that cover the reboot time.

From the SPcollects that captured prior the reboot:

SPX-cfg_extended.txt

navicli storagepool -type Thin -list
--------------------------------------------------------------------------------
Raw Capacity (Blocks):                522208988192
User Capacity (Blocks):               395698618368   <<<<<<<<<<<<<<<
Allocated Capacity (Blocks):          40619225088
Consumed Capacity (Blocks):           40619225088
EFD Cache State:                      Enabled

From the SPcollects that captured after the reboot:

Raw Capacity (Blocks):                522208988192
User Capacity (Blocks):               148438462464   <<<<<<<<<<<<<<<
Allocated Capacity (Blocks):          40619225088
Consumed Capacity (Blocks):           40619225088
EFD Cache State:                      Enabled

The storagepool reported wrong capacity is because the SPA panic triggered pool statistics to be reset and then recalculated. However the recalculate failed because of some FLUs were offline that caused the LU slice counts set to 0.

Verify in ktrace:
11:17:11.873          4 FFFFFA800ED01760 1  mlu:MLU I: mluPoolMgrListSlices[7198] ASYNC CALL returned SUCCESS for POOLOID 0x300000002, XID 5839
11:17:11.873         10 FFFFFA800ED01760 1  mlu:MLU I: MluPoolMgrAreAllFlusReady[9105] FlusReady=TRUE for Pool object: 0x300000002
11:17:11.873          4 FFFFFA800ED01760 1  mlu:MLU I: MluPoolMgrSendCheckingFlusReadyStatusMsg[228] Send Query SCN for pool 0x300000002
11:17:11.873         77 FFFFFA800ED01760 3  mlu:MLU E: MluPoolMgrSendCheckingFlusReadyStatusMsg[263] Peer down 5
11:17:11.873         41 FFFFFA800ED01760 3  mlu:MLU I: mluPoolMgrPoolObjectUpdateUserCapacity[7662] Pool object: 0x300000002 Current PoolUsableSizeInBlocks 395698618368 Settin    <<<<<<<<<<<<<<<<<<<<<<<
11:17:11.873          2 FFFFFA800ED01760 3  mlu:MLU I: mluPoolMgrPoolObjectUpdateUserCapacity[7662] g to 148438462464          <<<<<<<<<<<<<<<<<<<<<<<  pool userable size changed from 395698618368 to 148438462464
11:17:11.873         18 FFFFFA800ED01760 3  mlu:MLU I: MluObjMgrSetObjectStatus[2997] SET OBJ STATUS: Tr 0x16cf Obj 0x300000002 Status 0x0
11:17:11.873          4 FFFFFA800ED01760 3  mlu:MLU I: mluPoolMgrPoolObjectGotoReady[2432] Both SPs' SCNs are completed for Pool 0x300000002


Many LU slice count are zero.
11:17:08.383          5 FFFFFA800E525040 1  mlu:MLU I: mluPoolMgrFluObjectAddLuToSlicerSynchronous[3641] mluPoolMgrFluObjectAddLuToSlicer:SA SliceCount 0x0, BaseOffset 0x20000
11:17:08.383          5 FFFFFA800E525040 1  mlu:MLU I: mluPoolMgrFluObjectAddLuToSlicerSynchronous[3641] mluPoolMgrFluObjectAddLuToSlicer:SA SliceCount 0x0, BaseOffset 0x20000
11:17:08.384          5 FFFFFA800E525040 0  mlu:MLU I: mluPoolMgrFluObjectAddLuToSlicerSynchronous[3641] mluPoolMgrFluObjectAddLuToSlicer:SA SliceCount 0x0, BaseOffset 0x20000
11:17:08.385          5 FFFFFA800E525040 1  mlu:MLU I: mluPoolMgrFluObjectAddLuToSlicerSynchronous[3641] mluPoolMgrFluObjectAddLuToSlicer:SA SliceCount 0x0, BaseOffset 0x20000

Reboot SPB.