VxRail: Failed to Upgrade SKHynix SE5110 SSD Firmware when Upgrading VxRail Cluster to 7.0.410, 7.0.411 & 8.0.000

The following Failure on the VxRail UI plugin upgrade tab can appear to indicate a Failure running a firmware upgrade for SKHynix SE5110 SSD to version DZ02.
 

Error in lcm-web.log:

2022-12-29 12:10:24,317 ERROR [LCM] [lcm-node-0] c.v.l.s.LcmNodeUtilService [LcmNodeUtilService.java:578] SKHynix SSD Firmware for SE5110 Series bundle version in system is: DVCVersionInfo {version=DZ00, minBuild=} , failed to upgrade to target version: DVCVersionInfo {version=DZ02, minBuild=}

We can also see this firmware upgrade success log in the related node vxps_main.log:

2022-12-29 12:00:26,160 vxhttpsd     ThreadPoolExecutor-0_3 DEBUG    Current components on this platform are: {'101734': {'Version': ['6.0']}, '102303': {'Version': ['20.5.13', '20.5.13', '20.5.13']}, '102378': {'Version': ['20.5.13']}, '102573': {'Version': ['00.3D.67', '00.3D.67']}, '103710': {'Version': ['2.0']}, '104102': {'Version': ['2.52']}, '104299': {'Version': ['16.17.01.00']}, '104684': {'Version': ['4.1.0.0']}, '106883': {'Version': ['2.5.13.3024']}, '109356': {'Version': ['J004', 'J004']}, '109719': {'Version': ['1.12']}, '109911': {'Version': ['BA0D', 'BA0D']}, '110210': {'Version': ['DZ00', 'DZ00', 'DZ00', 'DZ00', 'DZ00', 'DZ00', 'DZ00', 'DZ00']}, '159': {'Version': ['2.16.1']}, '18981': {'Version': ['21.10.01']}, '25227': {'Version': ['6.00.30.00']}, '25806': {'Version': ['4301A66']}, '27763': {'Version': ['1.1.4']}, '28897': {'Version': ['6.00.30.00']}}.
2022-12-29 12:00:26,161 vxhttpsd     ThreadPoolExecutor-0_3 INFO     Post check for firmware upgrade passed.
2022-12-29 12:00:26,161 vxhttpsd     ThreadPoolExecutor-0_3 DEBUG    Reports of firmware upgrade process: {
  "TaskStatus": "Completed",
  "TaskMessage": "Firmware upgrade process completed successfully.",
  "OSRebootRequired": false,
  "Summary": [
    {
      "DUPName": "Serial-ATA_Firmware_8KVKG_WN64_DZ02_A00.EXE",
      "Type": "Firmware Update: SCSI Disk",
      "Status": "Completed",
      "MessageID": "PSINFO0001",
      "Message": "Job successfully completed with PercCLI.",
      "PercentComplete": null,
      "SupportedComponents": [
        "110210"
      ]
    }
  ]
}

But the DZ02 firmware upgrade has failed.

This impacts VxRail nodes that have SKHynix SE5110 with the following part numbers.

• SKHynix SE5110 RI 1.92 TB SATA SSD (HFS1T9G3H2X069N)
• SKHynix SE5110 RI 3.84 TB SATA SSD (HFS3T8G3H2X069N)

This problem was fixed from 7.0450 and the later versions in the 7.0 release, 8.0.100 and later versions in the 8.0 release. If the cluster cannot upgrade to the fixed version, check the below steps to work around this problem.

When encountering this upgrade failure, upgrade this firmware manually on a node per node basis.

Steps performed on a node per node basis in a serial method:

1. Download Serial-ATA_Firmware_8KVKG_WN64_DZ02_A00.EXE from dell support site, from the VxRail upgrade composite bundle or via the following link: https://www.dell.com/support/home/drivers/driversdetails?driverid=8kvkg 
2. Put the first impacted node into maintenance mode.
3. Open the iDRAC interface for the first impact node.
4. Navigate to Maintenance > System Update. Ensure that the Location is set to Local.
5. Click Choose File, select the Serial-ATA_Firmware_8KVKG_WN64_DZ02_A00.EXE file and click the Upload button.
6. On the bottom of the page select Install.
7. Click the Job Queue to verify the job was created.
8. In vCenter, right click the node you put into Maintenance Mode and select Reboot.
9. If prompted for a reason enter Drive Firmware Update.
10. Monitor the reboot to ensure that firmware upgrade succeeds.
11. Exit maintenance mode in vCenter.
12. Clear any alarms and verify system health before moving onto the next node.

For guidance on how to upgrade firmware via iDRAC:
Integrated Dell Remote Access Controller 9 User's Guide | Dell US

How to Update Firmware Remotely Using the Integrated Dell Remote Access Controller (iDRAC) Web Interface | Dell US

VxVerify 3.30.120 analysis for VxRail 7.0.372 at 2023-01-20 10:55 GMT. Profile 0-All checks.
#========================#=======#========#====================================================================#==============#
|  Hostname / Category   |Status  Dell_KB |  Warnings or Failures, unless tests Passed                         ; Product S.N. |
#========================#=======#========#====================================================================#==============#
| lab-dr-esxi-01         | Failure 207549 | idc_hwi: SKHynix SE5110 SSD requires manual firmware update                       |