We upgraded to OneFS 188.8.131.52 on June 25th. We started to get the Heartbeat alert emails daily for every node in the cluster. I tried to follow KB Article (000493496 - 8.0 Daily Hearbeat Alert and lowering frequency of it) but its not that accurate. I ended up changing the intervals to weekly but went into the GUI and just completely deleted the Heartbeat alert and Heartbeat channel out of the Alerts tab. But yesterday we received the emails again so something is still triggering it.
I ran the below commands and you can see the output below:
mycluster# isi event alerts view Heartbeat
Error in field(s): ID
Alert condition Heartbeat not found
mycluster# isi event settings view
Retention Days: 90
Storage Limit: 1
Maintenance Start: Never
Maintenance Duration: Never
Heartbeat Interval: weekly
Has anyone ran into this? Does anyone know how to stop the heartbeat completely?
Have you tried running isi event settings modify --heartbeat-interval <weekly, daily...etc> or you'd have to disable the alert channel it uses.
isi event alerts view Heartbeat works for me, our clusters are running 184.108.40.206
Channel: Heartbeat Self-Test
In OneFS 8.0 and later versions, cluster would generate heartbeat event every day.
The Heartbeat event belongs to Software events category. When configure Alert that uses SMTP channel for Software Events, the SMTP recipients would receive daily email notification for heartbeat event.
Use below step to stop receiving daily heartbeat events:
1. From WebUI, modify Alerts that utilize SMTP channel, and uncheck "Software Events".
2. Add all software event IDs except for heartbeat manually in CLI
# isi event alerts modify <alertname> --add-eventgroup= "100010043,400020001,400020002,400030001,400030002,400040002,400040005,400040008,400040009,400040010,400040011,400040012,400040013,400040014,400040015,400040016,400040017,400040018,400040019,400040020,400050001,400050002,400050003,400060001,400060002,400060003,400060004,400070004,400070005,400080001,400100001,400100002,400100003,400100004,400100005,400100011,400110001,400120001,400130001,400130002,400140001,400140002,400150001,400150002,400150003,400150004,400150005,400160001,400210001,400210002,499940001,499960001"
3. Send out a test heartbeat event to verify no more email alerts are generated.
# /usr/bin/isi_celog/celog_send_events.py 400050004
How to stop receiving email notification for HeartBeat event
seoeook, when I run an isi event alerts view alert2 for a list of alerts IDs, this is what I see:
mycluster# isi event alerts view alert2
Category: 100000000, 200000000, 300000000, 600000000, 700000000, 800000000, 900000000
Condition: NEW EVENTS
This is the only alert that uses SMTP that has Software Events checked off. So from that list which one is the Heartbeat one?
What the KB walks you through is removing the high-level categories of events and adding back all the individual events EXCEPT the heartbeat. You can't just remove an individual event from the default config, since that is defined with categories of events.
The categories you list (100000000, 200000000, etc.) each include many events.
The specific heartbeat event is 400050004, the one listed in the celog_send_events.py command-line.
In case it is useful for anyone, on 8.1 you need to specify the alert channel in order to add the eventgroups or you will get "Empty array field channel id".
In my case the name of the alert channel is "SMTP", and the alert name is "alert1":
isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > /var/crash/list.txt
isi event alerts modify alert1 --channel SMTP --add-eventgroup `cat /var/crash/list.txt`
It can also be done in a single command line, without creating a text file:
isi event alerts modify alert1 --channel SMTP --add-eventgroup `isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//'`