VNXe: SP panic due to "BMD buffer CRC mismatch" (Dell EMC Correctable)

SP may panic and cause FS to go offline.

Corruption is caused because of a very rare timing issue. 
Under some rare circumstances code hits a race condition while updating IB (Indirect blocks) and BMD CRC gets updated by 2 threads at same time and  hence causes this corruption. 
And also the expected iCRC in BMD Entry was different with entry in BMD Cache , so it reported BMD Cache mismatched.

A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 6: bmd_icrc:0x762bd332 bmd_crc:0xe04c355f bmd_owner:8338

When an FS/LUN comes online after hitting such a situation, attempt should be made to backup up the data ASAP. Avoid any further writes to that LUN/FS. There are high chances that LUN/can go offline during new writes on it.  FS/LUN would have been in offline state after hitting this issue.

Tech Support will need to run recovery to bring it online.  Please contact Dell EMC Technical Support or your Authorized Service Partner and quote this Knowledgebase article ID.

Triage logs:
-----------
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 6: bmd_state: 30 (version 0) flags:
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 6: ICRC_TRUSTED:SET
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 6: CRC_TRUSTED:SET
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 6: bmd_icrc:0x762bd332 bmd_crc:0xe04c355f bmd_owner:8338 <<<<<<<<< 
bmd_offset:4057293012993 bmd_totalWeight:261755 bmd_replcaId:64908 bmd_versionSetId:1 bmd_gen:32768 bmd_blockgen:0 bmd_seqnum=5874
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: UFS: 3: Unmounting fs 1073741844: Reason: BMD buffer CRC mismatch
A 05/02/21 06:58:50.014 cbfs 6fd7b701 c4_safe_ktrace CBFSA: CBFS Event Id 77579616268, Event Message: FsId: 1073741844 encountered a critical fault, 
action (PANIC_SYSTEM), FS CBFS Event Id 77579616268, Event Message: (File System taken offline (BMD buffer CRC mismatch)) is unmounted 
CBFS Event Id 77579616268, Event Message: internally.