Highlighted
2 Bronze

How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

We upgraded to OneFS 8.0.0.4 on June 25th.  We started to get the Heartbeat alert emails daily for every node in the cluster.  I tried to follow KB Article (000493496 - 8.0 Daily Hearbeat Alert and lowering frequency of it) but its not that accurate.  I ended up changing the intervals to weekly but went into the GUI and just completely deleted the Heartbeat alert and Heartbeat channel out of the Alerts tab.  But yesterday we received the emails again so something is still triggering it.

I ran the below commands and you can see the output below:

mycluster# isi event alerts view Heartbeat

Error in field(s): ID

Alert condition Heartbeat not found

mycluster# isi event settings view

     Retention Days: 90

        Storage Limit: 1

Maintenance Start: Never

Maintenance Duration: Never

Heartbeat Interval: weekly

Has anyone ran into this?  Does anyone know how to stop the heartbeat completely?

Thanks,

-Alan

Labels (2)
7 Replies
Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

Have you tried running isi event settings modify --heartbeat-interval <weekly, daily...etc> or you'd have to disable the alert channel it uses.

isi event alerts view Heartbeat works for me, our clusters are running 8.0.0.4

Name: Heartbeat

Eventgroup: 400050004

Category: -

Sev: *

Channel: Heartbeat Self-Test

Condition: ONGOING

Interval: Now

0 Kudos
Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

In OneFS 8.0 and later versions,  cluster would generate heartbeat event every day.

The Heartbeat event belongs to Software events category. When configure Alert that uses SMTP channel for Software Events, the SMTP recipients would receive daily email notification for heartbeat event.

Use below step to stop receiving daily heartbeat events:

1.  From WebUI, modify Alerts that utilize SMTP channel, and uncheck "Software Events".

2.  Add all software event IDs except for heartbeat manually in CLI

# isi event alerts modify <alertname> --add-eventgroup= "100010043,400020001,400020002,400030001,400030002,400040002,400040005,400040008,400040009,400040010,400040011,400040012,400040013,400040014,400040015,400040016,400040017,400040018,400040019,400040020,400050001,400050002,400050003,400060001,400060002,400060003,400060004,400070004,400070005,400080001,400100001,400100002,400100003,400100004,400100005,400100011,400110001,400120001,400130001,400130002,400140001,400140002,400150001,400150002,400150003,400150004,400150005,400160001,400210001,400210002,499940001,499960001"

3. Send out a test heartbeat event to verify no more email alerts are generated.

# /usr/bin/isi_celog/celog_send_events.py 400050004

KB 490436

How to stop receiving email notification for HeartBeat event

Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

seoeook, when I run an isi event alerts view alert2 for a list of alerts IDs, this is what I see:

mycluster# isi event alerts view alert2

      Name: alert2

Eventgroup: -

  Category: 100000000, 200000000, 300000000, 600000000, 700000000, 800000000, 900000000

       Sev: *

   Channel: smtp2

Condition: NEW EVENTS

This is the only alert that uses SMTP that has Software Events checked off.  So from that list which one is the Heartbeat one?

0 Kudos
Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

I do not understand the situation above.

For me, proceed as documented in KB 490436

Good.

Thank you

0 Kudos
Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

What the KB walks you through is removing the high-level categories of events and adding back all the individual events EXCEPT the heartbeat.  You can't just remove an individual event from the default config, since that is defined with categories of events. 

The categories you list (100000000, 200000000, etc.) each include many events. 


The specific heartbeat event is 400050004, the one listed in the celog_send_events.py command-line.

0 Kudos
Highlighted
1 Copper

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

In case it is useful for anyone, on 8.1 you need to specify the alert channel in order to add the eventgroups or you will get "Empty array field channel id".

In my case the name of the alert channel is "SMTP", and the alert name is "alert1":

isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > /var/crash/list.txt

isi event alerts modify alert1 --channel SMTP --add-eventgroup `cat /var/crash/list.txt`

It can also be done in a single command line, without creating a text file:

isi event alerts modify alert1 --channel SMTP --add-eventgroup `isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//'`

Cheers

Joseph

Highlighted
2 Bronze

Re: How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

Thanks Joseph. Running 8.2.1.0 and that worked beautifully.

0 Kudos