Isilon - Gen5 nodes reporting Pwr Unit Status: Power supply failed

Gen5 Isilon nodes can generate the following invalid power supply failures in events:

11.4521543 03/27 12:44 C    2    4527105        Pwr Unit Status: Power supply failed

Checking the isi_hwmon.log file on the node, there will be multiple archived isi_hwmon logs in the same day:

$ ls -l | grep hwmon
-rwxrwxr-x 1 7201552 1010    23329 Mar 27  2023 isi_hwmon.log
-rwxrwxr-x 1 7201552 1010     7978 Mar 27  2023 isi_hwmon.log.0.gz
-rwxrwxr-x 1 7201552 1010     7922 Mar 27  2023 isi_hwmon.log.1.gz
-rwxrwxr-x 1 7201552 1010     7876 Mar 27  2023 isi_hwmon.log.2.gz
-rwxrwxr-x 1 7201552 1010     7924 Mar 27  2023 isi_hwmon.log.3.gz
-rwxrwxr-x 1 7201552 1010     8512 Mar 27  2023 isi_hwmon.log.4.gz
-rwxrwxr-x 1 7201552 1010     7926 Mar 27  2023 isi_hwmon.log.5.gz
-rwxrwxr-x 1 7201552 1010     7949 Mar 27  2023 isi_hwmon.log.6.gz
-rwxrwxr-x 1 7201552 1010     7909 Mar 27  2023 isi_hwmon.log.7.gz
-rwxrwxr-x 1 7201552 1010     7930 Mar 27  2023 isi_hwmon.log.8.gz
-rwxrwxr-x 1 7201552 1010     7951 Mar 27  2023 isi_hwmon.log.9.gz

Contents of the isi_hwmon.log contains entries similar to the following:

2023-03-27T15:43:50.443412-05:00 <3.6>node-2(id11) isi_hwmon[2572]: Updating Event Instance ID: 774056190430235 Display ID: 11.4475931 msg: Pwr Unit Status: Power supply fai
led
2023-03-27T15:43:50.446432-05:00 <3.6>node-2(id11) isi_hwmon[2572]: Updating Event Instance ID: 774056190430236 Display ID: 11.4475932 msg: Pwr Unit Status: Power supply fai
led
2023-03-27T15:43:50.464123-05:00 <3.5>node-2(id11) isi_hwmon[2572]: SEL index 0004: Resetting: AC Power Loss
2023-03-27T15:43:50.464281-05:00 <3.5>node-2(id11) isi_hwmon[2572]: Resetting queues, ignoring all SEL entries before index=4
2023-03-27T15:43:50.473002-05:00 <3.6>node-2(id11) isi_hwmon[2572]: Unhandled exception caught in monitor_poll:
2023-03-27T15:43:50.479996-05:00 <3.3>node-2(id11) isi_hwmon[2572]: Exception caught during monitor_poll (line 2203): <type 'exceptions.AttributeError'>, 'QueryEvent' object
 has no attribute 'cancel'
2023-03-27T15:43:50.480270-05:00 <3.3>node-2(id11) isi_hwmon[2572]: Traceback (most recent call last):
2023-03-27T15:43:50.480337-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/usr/bin/isi_hwmon", line 2190, in monitor_poll
2023-03-27T15:43:50.480395-05:00 <3.3>node-2(id11) isi_hwmon[2572]:     handle_sel(sel_object, lastindex, newindex)
2023-03-27T15:43:50.480448-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/usr/bin/isi_hwmon", line 1705, in handle_sel
2023-03-27T15:43:50.480501-05:00 <3.3>node-2(id11) isi_hwmon[2572]:     result = handle_bmc_discrete(sel_detail)
2023-03-27T15:43:50.480553-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/usr/bin/isi_hwmon", line 794, in handle_bmc_discrete
2023-03-27T15:43:50.480605-05:00 <3.3>node-2(id11) isi_hwmon[2572]:     cancel_celog_discrete_events(events, entry)
2023-03-27T15:43:50.480656-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/usr/bin/isi_hwmon", line 741, in cancel_celog_discrete_events
2023-03-27T15:43:50.480708-05:00 <3.3>node-2(id11) isi_hwmon[2572]:     return celog.cancel_celog_events(events, sel_detail, celog_id)
2023-03-27T15:43:50.480759-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/b/mnt/obj/b/mnt/src/amd64.amd64/isilon/lib/python/stage-2.7/usr/local/lib/python2.7/site-package
s/isi/hw/hwmon/celog.py", line 490, in cancel_celog_events
2023-03-27T15:43:50.480820-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/b/mnt/obj/b/mnt/src/amd64.amd64/isilon/lib/python/stage-2.7/usr/local/lib/python2.7/site-package
s/isi/hw/hwmon/celog.py", line 484, in cancel_celog_event
2023-03-27T15:43:50.480925-05:00 <3.3>node-2(id11) isi_hwmon[2572]:   File "/b/mnt/obj/b/mnt/src/amd64.amd64/isilon/lib/python/stage-2.7/usr/local/lib/python2.7/site-package
s/isi/hw/hwmon/celog.py", line 366, in event_cancel
2023-03-27T15:43:50.480980-05:00 <3.3>node-2(id11) isi_hwmon[2572]: AttributeError: 'QueryEvent' object has no attribute 'cancel'

OneFS had changes made to CELOG that affected Gen5 hardware reporting.

OneFS 9.4.0.15 and above contain the fix for this issue.

If you are unable to upgrade the OneFS version, contact support and quote this document.