Avamar: Gen4S Hardware: How to address various disk events and errors

Events similar to the following events are observed in the UI or log files:

Controller ID: 0 Unexpected sense: PD = Port 0 - 3:2:n Information unit CRC error detected, CDB = 0x28 0x00 0xb6 0x77 0xc3 0x07 0x00 0x01 0xd0 0x00, Sense = 0x70 0x00 0x0b 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x47 0x03 0x00 0x00 0x00 0x00

Controller ID: 0 Unexpected sense: PD = Port 0 - 3:2:n Unrecovered read error, CDB = 0x28 0x00 0xd2 0x91 0xe0 0xf7 0x00 0x00 0x40 0x00, Sense = 0xf0 0x00 0x03 0xd2 0x91 0xe0 0xf7 0x0a 0x00 0x00 0x00 0x00 0x11 0x00 0x00 0x00 0x00 0x00

Controller ID: 0 Unexpected sense: PD = Port 0 - 3:2:n Hardware impending failure general hard drive failure, CDB = 0x03 0x00 0x00 0x00 0x40 0x00, Sense = 0x70 0x00 0x00 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x5d 0x10 0x00 0x00 0x00 0x00 

Controller ID: 0 PD Reset: PD = Port 0 - 3:2:n, Error = 3, Path = 0x5001234567890

Controller ID: 0 Unexpected sense: PD = -:-:n Recovered data - recommended rewrite, CDB = 0x2f 0x00 0x01 0x09 0x80 0x00 0x00 0x10 0x00 0x00, Sense = 0xf0 0x00 0x01 0x01 0x09 0x8f 0xed 0x18 0x00 0x00 0x00 0x00 0x18 0x06 0x00 0x80 0x09 0x19 

Controller ID: 0  Command timeout on PD: PD = Port 0 - 3:2:n No additional sense information, CDB = 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff, Sense = , Path = 0x5001234567890

Controller ID: 0 Unexpected sense: PD = Port 0 - 3:2:n Logical block address out of range, CDB = 0x2a 0x00 0xac 0x64 0xcd 0xe6 0x00 0x00 0x08 0x00, Sense = 0xf0 0x00 0x05 0xac 0x64 0xcd 0xe6 0x0a 0x00 0x00 0x00 0x00 0x21 0x00 0x00 0x00 0x00 0x00

Controller ID: 0  Unexpected sense: PD = Port 0 - 3:2:n Power on, reset, or bus device reset occurred, CDB = 0x2a 0x00 0x41 0xab 0x7a 0x00 0x00 0x00 0x80 0x00, Sense = 0x70 0x00 0x06 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 

Controller ID: 0 PD Predictive failure: Port 0 - 3:2:n

• • 3:2:0 - 3:2:11 or -:-:0 - -:-:11 are the internal standard disks
  • 3:2:12 or -:-:12 is the internal SSD disk.

See the resolution section for causes and actions.

1. Log in to the Avamar Utility Node as admin.

2. Elevate to root privilege.

3. Load the root keys per article Avamar: How to Log in to an Avamar Server and Load Various Keys. 

4. Using the information from the UI event or the dial-home Service Request, determine the node that produced the error message.

5. Connect to the node as root using the command below:

ssn 0.# --user=root

(Where 0.# is the physical node number) 

6. On the affected node:

a. Check the current status of the disks:

avsysreport pdisk

b. Review the /var/log/messages file for events relating to the disk reporting the event.

Required actions based on events logged:

• For the event "Recovered data - recommended rewrite":
  • If the disk reported no other events, this can be safely ignored
  • If there are other events such "Unrecovered Read Error," Contact Dell Technologies to determine if the disk must be proactively replaced.
• For the event "Logical block address out of range":
  • If the disk reported no other events, this can be safely ignored
  • If there are other events such "Unrecovered Read Error," Contact Dell Technologies to determine if the disk must be proactively replaced.
• For the event "Unrecovered read error":
  • If the event repeats more than three times on a single disk, the disk in question may have an integrity issue. Contact Dell Technologies to determine if the disk must be proactively replaced.
• For the event "Information unit CRC error detected":
  • If the disk reporting the event is 3:2:12, it indicates that the SSD is failing. Contact Dell Technologies to replace the node using the node transplant procedure.
  • If the disk reporting the event is 3:2:0-3:2:11, it indicates that the hard drive is failing. Contact Dell Technologies to proactively replace the disk.
• For the event "Hardware impending failure general hard drive failure":
• • If the disk reporting the event is 3:2:12, the SSD is failing (or has failed). Contact Dell Technologies to replace the node using the node transplant procedure.
  • If the disk reporting the event is 3:2:0-3:2:11, the hard drive is failing (or has failed). Contact Dell Technologies to proactively replace the disk.
• For the event "Predictive failure:
  • If the disk reporting the event is 3:2:12, the SSD reported a predictive failure caution through the SMART function, and indicates that the SSD is failing. Contact Dell Technologies to replace the node using the node transplant procedure.
  • If the disk reporting the event is 3:2:0-3:2:11, the hard drive reported a predictive failure caution through the SMART function, and indicates that the hard drive is failing. Contact Dell Technologies to proactively replace the disk.
• For the event "Power on, reset, or bus device reset occurred":
  • Review the /var/log/messages file to check if a single disk or multiple disks are reporting the error.
    • If a single disk is reporting the error, they may be related to the weekly consistency checks. Contact Dell Technologies to determine if the disk must be proactively replaced.
    • If multiple disks are reporting the error, the expander firmware is out of date, or the chassis is in a failed or failing state. Contact Dell Technologies to review the firmware levels or replace the node using the node transplant procedure.
• For the event "PD Reset":
  • Review the /var/log/messages file to check if a single disk or multiple disks are reporting the error.
    •  If the disk has been replaced recently, this message is expected and can be ignored.
    • If the drive has not been replaced and rebuilt on its own, the drive may be failing or failed. Contact Dell Technologies to proactively replace the disk.
• For the events "Command timeout on PD," or any other error not already mentioned Contact Dell Technologies to determine the next course of action.