Unsolved
This post is more than 5 years old
13 Posts
1
8375
How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?
We upgraded to OneFS 8.0.0.4 on June 25th. We started to get the Heartbeat alert emails daily for every node in the cluster. I tried to follow KB Article (000493496 - 8.0 Daily Hearbeat Alert and lowering frequency of it) but its not that accurate. I ended up changing the intervals to weekly but went into the GUI and just completely deleted the Heartbeat alert and Heartbeat channel out of the Alerts tab. But yesterday we received the emails again so something is still triggering it.
I ran the below commands and you can see the output below:
mycluster# isi event alerts view Heartbeat
Error in field(s): ID
Alert condition Heartbeat not found
mycluster# isi event settings view
Retention Days: 90
Storage Limit: 1
Maintenance Start: Never
Maintenance Duration: Never
Heartbeat Interval: weekly
Has anyone ran into this? Does anyone know how to stop the heartbeat completely?
Thanks,
-Alan
kbaryeh
5 Posts
0
July 10th, 2017 03:00
Have you tried running isi event settings modify --heartbeat-interval or you'd have to disable the alert channel it uses.
isi event alerts view Heartbeat works for me, our clusters are running 8.0.0.4
Name: Heartbeat
Eventgroup: 400050004
Category: -
Sev: *
Channel: Heartbeat Self-Test
Condition: ONGOING
Interval: Now
seoeook
3 Posts
2
July 10th, 2017 05:00
In OneFS 8.0 and later versions, cluster would generate heartbeat event every day.
The Heartbeat event belongs to Software events category. When configure Alert that uses SMTP channel for Software Events, the SMTP recipients would receive daily email notification for heartbeat event.
Use below step to stop receiving daily heartbeat events:
1. From WebUI, modify Alerts that utilize SMTP channel, and uncheck "Software Events".
2. Add all software event IDs except for heartbeat manually in CLI
# isi event alerts modify --add-eventgroup= "100010043,400020001,400020002,400030001,400030002,400040002,400040005,400040008,400040009,400040010,400040011,400040012,400040013,400040014,400040015,400040016,400040017,400040018,400040019,400040020,400050001,400050002,400050003,400060001,400060002,400060003,400060004,400070004,400070005,400080001,400100001,400100002,400100003,400100004,400100005,400100011,400110001,400120001,400130001,400130002,400140001,400140002,400150001,400150002,400150003,400150004,400150005,400160001,400210001,400210002,499940001,499960001"
3. Send out a test heartbeat event to verify no more email alerts are generated.
# /usr/bin/isi_celog/celog_send_events.py 400050004
KB 490436
How to stop receiving email notification for HeartBeat event
alanscopa2015
13 Posts
0
July 10th, 2017 06:00
seoeook, when I run an isi event alerts view alert2 for a list of alerts IDs, this is what I see:
mycluster# isi event alerts view alert2
Name: alert2
Eventgroup: -
Category: 100000000, 200000000, 300000000, 600000000, 700000000, 800000000, 900000000
Sev: *
Channel: smtp2
Condition: NEW EVENTS
This is the only alert that uses SMTP that has Software Events checked off. So from that list which one is the Heartbeat one?
seoeook
3 Posts
0
July 10th, 2017 17:00
I do not understand the situation above.
For me, proceed as documented in KB 490436
Good.
Thank you
gvaidman
20 Posts
0
July 11th, 2017 12:00
What the KB walks you through is removing the high-level categories of events and adding back all the individual events EXCEPT the heartbeat. You can't just remove an individual event from the default config, since that is defined with categories of events.
The categories you list (100000000, 200000000, etc.) each include many events.
The specific heartbeat event is 400050004, the one listed in the celog_send_events.py command-line.
_jvc73
1 Message
2
May 28th, 2018 05:00
In case it is useful for anyone, on 8.1 you need to specify the alert channel in order to add the eventgroups or you will get "Empty array field channel id".
In my case the name of the alert channel is "SMTP", and the alert name is "alert1":
isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > /var/crash/list.txt
isi event alerts modify alert1 --channel SMTP --add-eventgroup `cat /var/crash/list.txt`
It can also be done in a single command line, without creating a text file:
isi event alerts modify alert1 --channel SMTP --add-eventgroup `isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//'`
Cheers
Joseph
Murrieta
10 Posts
0
December 25th, 2019 14:00
Thanks Joseph. Running 8.2.1.0 and that worked beautifully.