ME4024 firmware upgrade = pool lost (model PX02SMF040 pn 0HKK8C)

I get this error for every Dell PX02SMF040 SAS SSD d/pn 0HKK8C drive:

An error was reported by a disk drive. (disk: channel: 0, ID: 16, SN: , enclosure: 0, slot: 16)(Key,Code,Qual,UEC:0x4,0x4C,0xA8,0x0000) 
(CDB:000000000000)(CmdSpc:0x0, FRU:0x0, SnsKeySpc:0x0)(Hardware, no decode for ASC/ASCQ)

To have all EIGHT fail at the same time, would be very unlucky

Wonder if this has anything to do with my issue....

Sadly even if I remove the drives from ME4 & plug into R740 with PERC H730P Adapter I get for each one:

Fault detected on drive x in disk drive bay 1

Controller recognizes them as:

Manufacturing Information 		
Part Number 	Information Not Available
Manufacturer 	TOSHIBA
Non RAID Disk Cache Policy 	Not Applicable
Product ID 	PX02SMF040
Revision 	A3AF
Serial Number 	Information Not Available
Manufactured Day 	Not Applicable
Manufactured Week 	Not Applicable
Manufactured Year 	Not Applicable

but cannot update the firmware, as the SAS-Drive_Firmware_69J5F_WN64_A3B4_A00.EXE claims

This Update Package is not compatible with your system configuration.

Full upgrade attempt log is here:

https://pastebin.com/UaJi9Fcm

I can hex edit the .fwh file changing the ID and then I get different error (Installed FW version: does not support FW update)

<Fri Oct 13 01:43:52 PM><Fri Oct 13 01:43:52 PM>DUPWrapper: Begin Device name=<Fri Oct 13 01:43:52 PM>DUPWrapper: checking Device 4 update ctrlid:0 deviceid:7
<Fri Oct 13 01:43:52 PM>DUPWrapper: after setting DeviceID
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice: Enter populateFileVersion
<Fri Oct 13 01:43:52 PM>ImageFiles::getCount count = 1
<Fri Oct 13 01:43:52 PM>getImage: Begin
<Fri Oct 13 01:43:52 PM>getImage: filepattern:*.fwh
<Fri Oct 13 01:43:52 PM>getImage: filename:[<Fri Oct 13 01:43:52 PM>KAI_PM2_Non-SED_A3B4.fwh<Fri Oct 13 01:43:52 PM>]  size<Fri Oct 13 01:43:52 PM>dd680<Fri Oct 13 01:43:52 PM>ImageFiles::DoesFilenameMatchTemplate: Begin
<Fri Oct 13 01:43:52 PM>DoesFilenameMatchTemplate:filename=KAI_PM2_Non-SED_A3B4.fwh filepattern=*.fwh
<Fri Oct 13 01:43:52 PM>DoesFilenameMatchTemplate:lowercase_filename=kai_pm2_non-sed_a3b4.fwh lowercase_filepattern=*.fwh
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: filepattern length = 5
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: wildcard index = 0
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: have last part
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: FPart=
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: LPart=.fwh
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: lowercase_filename.length()18
<Fri Oct 13 01:43:52 PM>ImageFiles:DoesFilenameMatchTemplate: pLPart.length()4
<Fri Oct 13 01:43:52 PM>getImage: End - Found
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice: Online Download Enabled
<Fri Oct 13 01:43:52 PM>m_D_Method = 1
<Fri Oct 13 01:43:52 PM>Image version = A3B4
<Fri Oct 13 01:43:52 PM>Min Image version = A301
<Fri Oct 13 01:43:52 PM>m_ProcessTime(byte) = x
<Fri Oct 13 01:43:52 PM>m_ProcessTime = x
<Fri Oct 13 01:43:52 PM># of devices = 4
<Fri Oct 13 01:43:52 PM>deviceID [0102241
<Fri Oct 13 01:43:52 PM>deviceName [0                              PX02SMF020
<Fri Oct 13 01:43:52 PM>deviceID [1102242
<Fri Oct 13 01:43:52 PM>deviceName [1                              PX02SMF040
<Fri Oct 13 01:43:52 PM>deviceID [200000
<Fri Oct 13 01:43:52 PM>deviceName [2                              PX02SMF080
<Fri Oct 13 01:43:52 PM>deviceID [3102244
<Fri Oct 13 01:43:52 PM>deviceName [3                              PX02SMB160
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice: Image version set: A3B4
<Fri Oct 13 01:43:52 PM>DUPWrapper: populate rc=f
<Fri Oct 13 01:43:52 PM>Comparing device ID:102241 with 00000
<Fri Oct 13 01:43:52 PM>Comparing device ID:102242 with 00000
<Fri Oct 13 01:43:52 PM>Comparing device ID:00000 with 00000
<Fri Oct 13 01:43:52 PM>depCheck: Found matching device
<Fri Oct 13 01:43:52 PM>Comparing device ID:102244 with 00000
<Fri Oct 13 01:43:52 PM>toLowerCase: begin
<Fri Oct 13 01:43:52 PM>toLowerCase: begin
<Fri Oct 13 01:43:52 PM>toLowerCase: begin
<Fri Oct 13 01:43:52 PM>toLowerCase: begin
<Fri Oct 13 01:43:52 PM>depCheck: Installed FW version: does not support FW update
<Fri Oct 13 01:43:52 PM>depCheck: Min required version:A301
<Fri Oct 13 01:43:52 PM>depCheck: Min version check failed
<Fri Oct 13 01:43:52 PM>DUPWrapper: error from depCheck:13
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 1
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 2
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 3
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 4
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 5
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 6
<Fri Oct 13 01:43:52 PM>XMLWriter::addUpdateDeviceInfo: 7
<Fri Oct 13 01:43:52 PM>DUPWrapper:No devicea updated.
<Fri Oct 13 01:43:52 PM>DUPWrapper:Deleting ImageLoader.
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice::~SASHardDriveDUPDevice::Deleting m_PR_stopped
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice::~SASHardDriveDUPDevice::Finished deleting m_PR_stopped
<Fri Oct 13 01:43:52 PM>SASHardDriveDUPDevice::~SASHardDriveDUPDevice::Deleting m_PR_disabled
<Fri Oct 13 01:43:52 PM>ImageFiles::Finished deleting m_PR_disabledr
<Fri Oct 13 01:43:52 PM>swrSASHardDriveDUPDevice::~swrSASHardDriveDUPDevice::Deleting m_BAS_disabled
<Fri Oct 13 01:43:52 PM>swrSASHardDriveDUPDevice::~swrSASHardDriveDUPDevice::Finished deleting m_BAS_disabledr
<Fri Oct 13 01:43:52 PM>swrSASHardDriveDUPDevice::~swrSASHardDriveDUPDevice::Deleting m_config_info
<Fri Oct 13 01:43:52 PM>swrSASHardDriveDUPDevice::~swrSASHardDriveDUPDevice::Finished m_config_info
<Fri Oct 13 01:43:52 PM>DUPWrapper: Checking for process time delay
<Fri Oct 13 01:43:52 PM>DUPWrapper: deviceProcessWaitTime=0
<Fri Oct 13 01:43:52 PM>DUPWrapper: AtLeastOneDeviceUpdatedSuccessfully=0
<Fri Oct 13 01:43:52 PM>DUPWrapper: After takeAction +++++++++++
<Fri Oct 13 01:43:52 PM>DUPWrapper: takeAction complete

Anybody ANY idea how to force upgrade the firmware?

Nobody at all? Some clever minds around, surely somebody has an idea?

https://www.dell.com/community/en/conversations/powervault/toshiba-px02smf080-has-lost-communication-on-the-same-date/647fa292f4ccf8a8de821fed

Hello,

Could you pull the ME4 support logs and upload for us to get a look?

How to Collect support logs on ME4 models

https://dell.to/4bk7aHm

How to Share Log Files, Screenshots, and Error Messages with Dell

https://dell.to/4bk7beo

I will send you a private message with a case number you can upload to.

@CJUMag 

Honestly, whey did I not think about it? It is so simple.

Well, I did think about it & even tried, but seems that does not work with RAID controller, only work with plain HBA.

So had to buy cheap SAS HBA and cable to connect SAS drive to it and indeed the updater did update the drive... after drive (good I only had 8, so only needed to poweroff/switch drive/power on 8 times

Drives back to the array in same spaces (they were sticky note numbered), a bit of complain from array

But the volume appeared & Healthy

Now the tricky part - rename the volume (because I had (maybe did not have to, but did) to create small volume on the existing DG with identical name. Reboot controllers (so they can see all volumes), I used the opportunity to upgrade the firmware on the array at the same time - 20-30 min is no lie in the docs! - https://www.dell.com/support/kbdoc/en-uk/000120120/me4-how-to-upgrade-controller-firmware

After all up made all the mapping to the FC hosts again

Still could not use the disk (showing in disk management) in Hyper-V Cluster.

Obviously there was some stale data & had to use

Clear-ClusterDiskReservation –Disk 4

At this point could add the disk to cluster & add it to CSV and ... bingo!

All data accessible as if nothing ever happened!

Just in case I backed it up elsewhere (but I expect these drives will now work forever (they still show 98% life left)

So thanks again!

Seb

@SPGS​ Hi, facing a similar issue with PX02SMF040 drives. Tried to update the firmware using a Dell R640 with PERC H730 mini in HBA mode without success. The update process ends with a weird error: "The operation failed due to an invalid image file". Observing the update logs, it seems that the new firmware is being sent to the drive via writeBuffer commands, but the last write fails. Despite the failure, the drive now seems to be updated to version A3B4, but still doesn't operate - 0 size and in a bad state.

Reading your last post, I wonder whether it's somehow the controller's fault. Can you please share the exact controller model number that you used for updating your drives? Perhaps can you please also share a link to the item listing where you purchased it? Many thanks!

Hi,

You might be facing a compatibility firmware issue, I've check in the downloads for A3B4 https://dell.to/3QCUFhH; the firmware released is for 12G and 13G servers. Do you have any of the servers that you can install the drives in? - which is similar to SPGS's shared solution by using a HBA card. This might work cause it is compatible. 

@DELL-Joey C​ 

Hi Joey, thanks. As said, I'm using a Dell R640 server with a PERC H730 mini controller set to HBA mode. I'm using the updater from the link you provided. Unlike @SPGS case, according to the logs, the update DOES start, and the firmware IS being sent to the drive. Indeed, the firmware version of the drive was updated afterwards. However, the updater failed, and the drive is still non-operational. Since the data is important, I thought that like @SPGS, maybe using another controller could aid in getting a successful update of the drives. Hence I'm interested in the exact model number and perhaps even a purchase link. Thanks!

Many many thanks @SPGS !!

Just to make sure - were you able to use the plain LSI 9211 controller, with the regular dell updater exe file?

Also, out of curiosity and hope - have you tried, and failed, to update using a raid controller? Like in my case? In which the drive partially updates but still doesn't work afterwards? I'll try with an HBA controller, but it would be very encouraging to know that someone was in my exact situation and was able to overcome the problem..

Again, thanks a lot!