R415 - Intermittent kernel, chipset , PCI parity and CPU machine checks errors - RHEL 6.4

 
Issue:

R415 Servers runing RHEL 6.4 may experience a random hang.  This typically occurs after the kernel "uhuh..." message.  The system will recover from the hang, but it will likely occur again.


Solution:

Update the BIOS to 1.2.4 or newer.


Additional Information:
 
Check the for the following symptoms: 

 • Hardware log:
 A PCI parity error was detected on a component at bus 0 device 0 function 0.
 CPU 1 machine check error detected
 CPU 2 machine check error detected 

 • /var/log/messages shows:  
 kernel: Uhhuh.  NMI received for unknown reason 21 on CPU 0.
 Kernel: Do you have a strange power saving mode enabled?
 Kernel: Dazed and confused, but trying to continue
 
 • After iDRAC boot up (handed off to the OS):
 Chipset Error interrupt at 1000:ECDD.
 Type (S)hut off NMI, (R)eboot, other keys to continue

Screenshots:

 








Check /etc/grub.conf and search for following kernel parameters:
acpi=off
noapic

Example with both listed:
default=0
timeout=5
serial --unit=1 --speed=115200
terminal --timeout=5 serial console
title Red Hat Enterprise Linux (2.6.32-358.el6.x86_64)
 root (hd0,0)
 kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_cl341-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us console=ttyS1,115200n8 LANG=en_US.UTF-8 rd_LVM_LV=vg_cl341/lv_root rd_LVM_LV=vg_cl341/lv_swap acpi=off rd_NO_MD noapic crashkernel=auto SYSFONT=latarcyrheb-sun16 rd_NO_DM
 initrd /initramfs-2.6.32-358.el6.x86_64.img

If any of these are present, make a backup copy of grub.conf:
cp /etc/grub.conf /etc/grub.backup

Remove both entries.

Example of a default grub.conf without these entries:
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux (2.6.32-358.el6.x86_64)
 root (hd0,0)
 kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/VolGroup00-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap rd_LVM_LV=VolGroup00/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup00/lv_swap rd_NO_DM rhgb quiet
 initrd /initramfs-2.6.32-358.el6.x86_64.img
 
 
This will resolve issue if the symbolic link to /boot/grub/grub.conf exists. To check this...
[root@localhost ~]#cd /etc
[root@localhost etc]#ls -l
lrwxrwxrwx.  1 root root   22 Aug 22 09:30 grub.conf -> ../boot/grub/grub.conf

If this link is not there, run this command.
[root@localhost etc]#ln -s /boot/grub/grub.conf grub.conf