[PowerScale] F600: Kioxia CD5 7.68TB NVMe SSD may fail prematurely when running out-of-date firmware

Summary: An issue has been identified in Kioxia CD5 7.68TB NVMe SSD firmware versions prior to 1.1.5 that could cause the drive to fail prematurely. This failure mode can in some cases cause the node to panic and reboot if it is running OneFS 9.1.0.4 or older. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

An issue has been identified in Kioxia CD5 7.68TB NVMe SSD firmware versions prior to 1.1.5 that could cause the drive to fail prematurely. If the node is running OneFS 9.1.0.4 or older, this failure mode can in some cases cause the node to panic and reboot, recording a kernel panic message similar to the following in the system log: 
panic @ time 1606735946.256, thread 0xfffff802cab45000: bio 0xfffff802cb64e060 cmd 1<BIO_READ> stuck in iosched nvd2 for over 240 seconds (240 seconds, cycle 40752ea883a294, arr 4074b401850eac), total_inprog 4, bio_rd_inprog 0, bio_wr_inprog 4, bio_rd_inqueue 1, bio_wr_inqueue 0

Cause

The drive manufacturer has identified an issue in the shipping firmware for this drive that can lead to premature drive failure. DellEMC PowerScale Engineering has identified an issue in the OneFS code that leads to this failure mode causing a kernel panic, rather than being handled gracefully.

Resolution

To resolve the drive firmware issue causing premature failure of these drives, update the drives to firmware version 1.1.5 or newer by installing the latest Drive Support Package (1.37.2 or newer) and running a drive firmware update. The drive firmware process is nondisruptive and can be run without scheduling a maintenance window. Instructions for installing a new Drive Support Package (DSP) and updating the drive firmware on your cluster can be found in the Release Notes document accompanying the DSP release.

To resolve the OneFS issue causing PowerScale nodes to panic in certain drive failure scenarios on OneFS version 9.1.0.4 or older, install the latest OneFS Roll-Up Patch (version 9.1.0.6 or newer), or upgrade to OneFS 9.2 or newer. Installation instructions can be found in the Release Notes document accompanying the latest Roll-Up Patch (RUP) or OneFS release.

Affected Products

PowerScale F600
Article Properties
Article Number: 000186477
Article Type: Solution
Last Modified: 05 Aug 2021
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.