iDRAC6 network stops responding

Question

Hi. We have a bunch of R710's all with iDRAC6. Some have the dedicated interface, some have a blank and share with LOM1. All have worked flawlessly for several years and all are running firmware 2.90. All servers are kept up to date as per the catalog via OpenManage Essentials. In the last month we have suddenly started to experience network issues with the iDRAC on a number of servers. The primary symptom is that the iDRAC network stops responding, the web interface is offline and ping ceases which is then picked up by our monitoring tools. A physical examination of the server shows the network interface is up and active, the iDRAC controls on the front panel work perfectly (and show the non-responsive IP address configured) and no alarms are logged in the Dell logs. To resolve the issue: If the server is rebooted, the iDRAC starts responding again. If racadm racreset soft is run from the Windows command line, the iDRAC starts responding again (without reboot of the physical box) The common factor is that it is the servers that share LOM1 that are affected, the servers with the dedicated interface installed are unaffected by this phenomenon. All affected servers have BCM5709C on board interfaces running QLogic driver 7.13.54.0 (reported by Windows. Actual Dell package is QLogic BCM 57xx and 57xxx Driver Family 20.0.0) with firmware 8.7.26. As the chances of a whole bunch of iDRACs going bad at the same time are slim to none, and only the iDRACs with shared LOM are affected, what are the likely candidates? As the shared iDrac piggy backs on interface 1 onboard, I assume the most likely candidates are: Windows update conflict with something installed recently. All servers are kept up to date so would be likely something installed within the last two months. Network firmware/driver issue. No later versions being offered by OpenManage Essentials. Thanks in advance

DELL-Josh Cr · Answer

Hi,

Can you private message me the service tag so we can get some additional information?

What OS are the servers running? Are you running racadm racreset from the OME server or from the server with the iDRAC?

APweb · Answer

Hi Josh All servers are running Server 2012 R2 Datacentre. I am running racadm racreset from the Windows command line. I will PM you a few service tags. Thanks!

DELL-Josh Cr · Answer

Does racadm getsysinfo return info about the iDRAC while it is not responding?

APweb · Answer

I will need to wait for the next incident and test. I will get back to you.

APweb · Answer

Hello Josh

Just having an incident now, this time on an R210 II

racadm getsysinfo is hanging

Drac pings remotely but web interface is not responding

Windows SNMP service cannot be controlled. After terminating and restarting, remote monitors start responding but Dell specific monitors such as Poweredge Hard Disk and System Health show not responding (using PRTG for monitoring and Dell OpenManage Essentials)

DSM SA Data Manager service cannot be controlled (it self terminated when attempting to stop, hanging on start)

APweb · Answer

Another incident, R610.

All remote monitoring went offline. Physical examination of the server (luckily an engineer was at the datacentre) showed the status screen frozen on 170W (power reading). Holding down i button for 20 seconds did not reset. Racadm getstatus hung.

Server shutdown. All lights off except bizarrely the status screen which continued to show blue and 170W.

Removed power for 2 minutes, restarted, iDrac re-initialised.

There is an inherent problem with this iDRAC release across multiple models.

APweb · Answer

More incidents recently including R510, R710 and R610.

I currently have an iDrac frozen R710 if it would be useful to collect any data. According to our remote monitoring, the response time of the iDrac web interface increases more and more until it freezes and stops responding.

This is the device that is currently frozen. If I attempt a racadm racreset soft, I get the following (after about 2 hours delay):

ERROR: A firmware update is currently in progress. Unable to reset the
RAC at this time.
===============================================================================
IMPORTANT NOTE!
The RAC is unable to communicate with the BMC. This condition may
occur because of (1) no BMC is present, (2) missing or disfunctional
IPMI-related software components. Many RAC features depend on BMC
connectivity in order to work properly, and you may see failures
as a result.
===============================================================================

There is no firmware update in progress nor has there been on this server recently.

This problem is incredibly frustrating because it:

A) Stops all remote SNMP monitoring of the server

B) Requires a datacentre visit to physically remove power from the server for 3 minutes to resolve.

Holding down the front iDrac button does not initiate a reset. The iDrac is completely frozen. So frozen that when you power off the server and all the front lights go off, the iDrac display remains illuminated with the last value shown.

PowerEdge Hardware General

Was this post helpful?