PowerFlex 3.x Rebuild Does not Complete While A Storage Pool Is Degraded

Scenario
The rebuild operation is not progressing while a storage pool is degraded because the MDM has received conflicting information about the current state of the affected comb/s in that storage pool.

Symptoms
For this example, there is a single comb that is in a NORMAL_NEEDS_REBUILD state but the rebuild activity would not complete. (In other cases, there could be many combs in this state)

MDM events showing that the cluster is in a degraded state due to a device failing:

2021-07-20 12:02:19.674 SDS_DEV_ERROR_REPORT      ERROR    	 Device error reported on SDS: sds01, Device: /dev/sdu. State: NORMAL upDownState: UP processState: DEV_ERR_INPROGRESS 
2021-07-20 12:02:20.785 MDM_DATA_DEGRADED         ERROR    	 The system is now in DEGRADED state.

The multihead showing the comb is in an "NR" state (NEEDS_REBUILD). The device holding the SEC chunk of data is DEGRADED_FAILED.

912: State: NORMAL  Task: NONE  Step: NONE  PTask: ROLLBACK_MIGRATE  Trig: HARDEN_COMPLETE
      packedState: [__,__,__,NR,__,__,__,__,__,PR,__,1,NON] NOT-BUSY priTgtGen: 132901 migrateGenNum: 32     grep -e 291680238390 -e "MultiHead: 522d0047 Row: 912 "
      PendTasks:   [__,__,__,__,__,__,__,__,__,__,__,__,__] PriDevId: e5ffe96500170012
       C0: [PRI,_,_,_,_,_,2]  [_,_,_] DevId: e5ffe96500170012 mdmTgtGen: 3189, tgtCombGenNum: 1, capFlags:DEGRADED_HEALTHY | PRIMARY (0x48)
       C1: [SEC,_,_,_,_,E,0]  [_,_,_] DevId: e5fde9be0016000f mdmTgtGen: 3195, tgtCombGenNum: 2, capFlags:DEGRADED_FAILED (0x10)
      MIG: [FRE,_,_,_,_,_,0]  [_,_,_] DevId: 0000000000000000 mdmTgtGen: 0, tgtCombGenNum: 0, capFlags: (0x0)

Primary comb holding SDS trace logs showing the comb going into a NORMAL_NEED_REBUILD state because the secondary comb-holding device had a Partial Device Error (PDE) event:

 20/07 12:02:19.690499 0x7fe648596db0:raidComb_SetCombState:00129: Set comb raid state 291680238390, raid state 0x1 => 0x40 (NORMAL => NORMAL_NEED_REBUILD) stateReason: PDE_CORRUPTION_ORIGIN_SECONDARY

Impact

The cluster is in a degraded state, and the data is not protected.

The secondary comb holding SDS had a device go into an error state and also had a PDE event on the same device. These two states do not go together which will cause the rebuilt operation to not progress.

Determine the faulty SDS device ID by running the following command:

 scli --query_properties --object_type DEVICE --all_object --properties INACCESSIBLE_CAPACITY_IN_KB,EXPOSED_CAPACITY_IN_KB

Clear the PDE flags by running the below command using the device ID:

 scli --test_inaccessible_data --device_id <id>

This should clear the PDE, and the rebuild should progress.

In the case INACCESSIBLE_CAPACITY_IN_KB is 0.

- Put the SDS having the Faulty SDS device in Instant Maintenance or Protected Maintenance mode. This should help complete the rebuild for the comb shown above.

- When you exit SDS Maintenance mode, use the flag --force_failed_device_id <device_id> against the failed device.

Impacted Versions
PowerFlex 3.5.x

PowerFlex 3.6.x

Fixed In Version
PowerFlex 3.6.2