Alerts - Server Off

Hi Ryan,

Can you post a screen shot or the text of the Power SNMP alert from the SNMP console in OME?  That might give us some clue on why the email rule did not trigger.

Thanks,

Rob

delltechcenter.com/ome

So in the fourth screenshot, the server highlighted in blue, gws01, is the server that was off this morning but no message was sent.  I was able to send a test message to the email address and it was received. (I did removed the email addresses from the screenshots.)

Ah, I apologize, I misunderstood.  Looking through the console alerts, there is no alert saying that the server is down.  It shows a controller information alert Saturday morning and then a SNMP Agent cold start message this morning when it was powered back on.  OME never generated an alert for this event.

Is this something I need to configure in OME?

Yeah, Ok, but can you post a screen shot of the actual _alert_ that came in to the SNMP alert console in OME itself?  That alert would have the OID, IP address, etc.

Thanks,

Rob

Yes, it was a power outage.  Likely the server did go down gracefully since it's on a UPS and software that will power the server down after an amount of time.  However, I would really like to have a way to receive an alert that shows when a server is reported as being in an off condition in the console.  Is this possible?

I appreciate your feedback.

Oh...well there's your problem :) :)

I think if the server "up and dies" then there is no outgoing SNMP alert from the crashed box.  The email gets sent as a filter on the incoming SNMP alerts.  No SNMP alert, then no email.

There are some predictive / redundancy failure events, but that depends on the cause of failure.

Do you know why it went down?

Rob

Ok, is your UPS set up to send SNMP alerts when it kicks in and does a shutdown?  This would be something outside the OMSA or iDRAC instrumentation.  OME does support MIB import.  So if there is a way to do this with your UPS and it has a MIB, that might be a way to do this.

Thanks,

Rob

Another way to achieve this in OME would be through internal connection status alerts. OME generates internal connection status alerts whenever it detects change in connection status of the device (On to Off or vice-versa). Internal connection status alerts can be enabled through OME preferences. Navigate to Preferences --> Alert Settings, select the required check box and apply settings as shown below:

Internal connection status alert looks like this:

Since you have selected all the alert categories and severity for the email action, this should trigger an email when the server is reported down in the subsequent discovery operation or status poll.

Hope this helps.

Ok, I've enabled the Internal Connection Status alerts.  I'll see if there is a box I can setup to test this with.  Thank you for the suggestions, I really appreciate it.

We're getting new PowerEdge T420's for all our school locations so I want to make sure OME is working properly before this is done (because in theory we should have no problems with our new servers right...). 

Ok, I found a server that I could power off and I did receive two alerts to my email (I removed identifiable names and tag numbers).

Device:******************* **************, Service Tag:**********, Asset Tag:, Date:12/9/2014, Time:09:13:34, Severity:Warning, Message:LinkDown Port or number: 14

and

Device:******************* **************, Service Tag:***************, Asset Tag:, Date:12/9/2014, Time:09:13:36, Severity:Warning, Message:Device *************** has changed status to Unknown.

My last question, is there a way for the status to report that it's off instead of Unknown?  This is certainly better than nothing but Unknown isn't specific even though via the console the Status says Off.

Ok, just to clarify...when you click on the down server, then you are saying the health is unknown (on the tree)  and the connection status is showing off...is that correct?

Rob

For this I imported the UPS MIBs and then monitored the UPS. This alerted us to a problem with power to the server through the UPS. It's highly unlikely there would be a situation where the server and UPS would die at the very same moment.

Wow, I'm definitely going to look into this.  Thank you.

The connection status alert (reporting system down) has higher priority than that of health status alert (reporting unknown health). So, when the server goes down, irrespective of whether discovery or status poll has been performed, alert reporting system is down should ideally be received first.

Can you try the use case once more?

1. Discover the server.

2. Shut it down.

3. Run discovery or status poll.

4. Observe the alert log.

We would like to make sure that the behavior you observed is consistent. In such case, we would like to take a deeper look into the OME setup that you have. You can open a support ticket based on this result.