Start a Conversation

Unsolved

S

1 Rookie

 • 

8 Posts

21

February 22nd, 2024 17:15

Predicted drive failure on VRTX

Hi, 

I saw the predicted drive failure error pop up on my VRTX. I already swapped out drive 9 but didn't know there was a recommendation to force the drive to Offline status. Does that mean I need to make the VDI offline? I'll need to schedule downtime since these servers are in production. I ran the TTY log from the controller and searched for "puncture" and didn't find any listed in the log. 

1 Rookie

 • 

8 Posts

February 22nd, 2024 20:57

Update: Ran a CRC check and disk 9 came back fine. No longer flagged with failure predicted. I'm still getting the SCSI Sense error. Is that an indication that disk 0:0:14 will need replacing soon?

Moderator

 • 

3.2K Posts

February 22nd, 2024 21:50

Hello,

 

Good to see it is no longer flagged as failure predicted.

 

Next I would make sure that the firmware for Disk and PERC controller be updated if it is not already.

Then keep an eye on it. It is not certain it would go pred fail but monitor it.

Some Sense errors are normal to have and the disk will remap bad blocks.

Now is you are getting a lot of them then you can consider replacing it.

1 Rookie

 • 

8 Posts

February 22nd, 2024 23:09

@DELL-Charles R

Logs are filled with it. I guess it's time to replace. 

Moderator

 • 

3.6K Posts

February 23rd, 2024 00:01

Hello I second that. Failure predicted.

Respectfully,

February 23rd, 2024 18:34

Hi, 

Looks like after replacing disk 9 the system is showing a VDI degraded message. Looks like a rebuild is taking place. How long does the rebuild typically take? 40.92TB VD. RAID 6 25x 1.8TB 10K SAS drives. I'll wait until the rebuild is done before proceeding with replacing disk 14.

Looks like CRC Check stalled out at 18%. Guess it's queued up while the rebuild is in progress or it kicked off again?

Moderator

 • 

3.2K Posts

February 23rd, 2024 18:52

40TB can take a long time to rebuild.

Yes please wait for the rebuild to complete before swapping out any other drives.

We shouldn't have a consistency check running during a rebuild. I don't know of a way to stop it and it may be paused during the rebuild as you suspect.

 

Let the rebuild complete, the progress percentage can be slow to change for a large VD.

After the rebuild completes see if the consistency check starts again.

I would recommend another consistency check after disk9 completes rebuild and before changing any other drive.

 

February 28th, 2024 16:42

Rebuild of Disk 9 completed. Swapped in a replacement for Disk 14. Rebuild started but still seeing Stor2 events. More PDR54 though so maybe it's correcting those blocks. 

Moderator

 • 

3.2K Posts

February 28th, 2024 16:50

Once disk 14 completes you may run a Consistency check again.

1 Rookie

 • 

8 Posts

February 29th, 2024 17:35

Hi, 

An update:

Swapped out with a new disk 14. Rebuild kicked off. When logged in yesterday it showed Slot 14 was empty along with a VDI degraded error. Not sure what to do so I swapped in with another new disk unsure if it was a compatibility issue or not. No VDI degraded error this morning but it's still showing slot empty for disk 14. Chassis log is showing activity for Disk 14 though! Lights are physically blinking and sync'd on disk 14 on the chassis. VDI is in optimal state according to the chassis log. Attached the storage controller logs for reference. Could this just be the CMC not showing the right status? Even shows 25 disks installed. 

TTY Logs:

RAID.ChassisIntegrated.2-1 (2).log

RAID.ChassisIntegrated.1-1 (2).log

(edited)

Moderator

 • 

3.2K Posts

February 29th, 2024 19:40

Hello,

 

That is strange it is not showing in the slot but you have LED activity.

Note I did only count 24 disk on the image showing all the drives.

 

Try to get a maintenance window and Restart the VRTX: Try restarting the VRTX enclosure to see if it resolves the issue. This can help refresh the system and reestablish the connection with the disk.

Verify that the SAS cables connecting the disks to the controller are securely attached.

 

Update firmware and drivers: Ensure that the VRTX enclosure, disk controller, and any related firmware and drivers are up to date. Outdated firmware or drivers can sometimes cause compatibility issues.

 

If it is still not reporting the drive is installed then gather a new log and upload to the information I will Private Message you.

 

I could not get the logs you posted, it is blocked from access so I will provide information to upload the log.

No Events found!

Top