Start a Conversation

Unsolved

This post is more than 5 years old

15411

November 22nd, 2016 12:00

Isilon x410 Nodes - Front Panel Lights

So this is kind of driving me crazy.  Whenever something triggers a critical alert, my front panels go amber, maybe on the particular node or for all of them.  I clear whatever was causing the issue (yesterday it was planned firmware and patch updates to all nodes) but the front panels never clear the amber light.  My WebGUI is back to green, but my front panels are amber.  The biggest problem is this is in a datacenter on the other side of my campus, and the night crew does a walk-through and will often alert me that "something's wrong with the Isilon".  We're on x410 nodes.  I clear the amber light by reseating the front panel.

I am using the WebGUI to clear alerts, and haven't tried the CLI yet.  We're on OneFS 8.0.0.2 and I'm still finding quirks between WebGUI and CLI so I suppose that could be contributing to it.  Does anyone else experience this?  Is there any way for me to clear the front panel (or even know it is amber) if my WebGUI is green and isi status is all OK?

5 Practitioner

 • 

274.2K Posts

June 15th, 2017 14:00

All,

I tested this issue and there is something going on with the alerter. Please open a service request with Technical Support and reference bug 195774. There is a workaround and it is to remove the event ID flags under:

/ifs/.ifsvar/db/celog_analysis/led

Easy command to do all nodes and all events: (This only clears the LED state, your actual CELOG alerts are untouched.)


for i in $(isi_nodes %{id}); do rm -r /ifs/.ifsvar/db/celog_analysis/led/$i/*; done

Note that the led folder is not based on the LNN of the node but rather the node ID, which is unique and never reused.

26 Posts

June 22nd, 2017 22:00

Hi alandelgado,

Thanks for your information how to resolve this issue.

I have a question.

How can I specify the node ID???

5 Practitioner

 • 

274.2K Posts

June 27th, 2017 08:00

The node ID is specified via the loop by the $i variable. If you wanted to find out the ID of all your nodes, run this command from any node:


isi_nodes %{name} %{lnn} %{id}

Then you will see the ID to LNN mapping and choose appropriately. For example:

root@cluster1-1:~# isi_nodes %{name} %{lnn} %{id}

cluster1-1 1 1

cluster1-2 2 2

cluster1-3 3 3

cluster1-4 4 4

cluster1-5 5 5

cluster1-6 6 6

cluster1-7 7 7

cluster1-8 8 8

cluster1-9 9 9

cluster1-10 10 10

cluster1-11 11 11

cluster1-12 12 12

cluster1-13 13 13

cluster1-14 14 14

cluster1-15 15 15

26 Posts

June 30th, 2017 01:00

Hi alandelgado,

Thanks for your reply. I got it.

1 Rookie

 • 

62 Posts

August 31st, 2017 16:00

So sounds like patch-197312 released Aug 25 may address this issue? It's only for v8.0.0.4 currently though.  Since posting this issue we have since upgraded to v8.0.0.4 but given how minor this is right now, I'm not going to install it unless another more critical patch comes along that I need.  But then again, I only have 4 nodes to re-seat any time this happens, I might feel differently if I had 100!  If anyone is feeling brave, install this patch and report back the results.

Ryan

7 Posts

September 1st, 2017 03:00

Issue didn't recur for me so far across reboots. But we're on 8.0.0.5

alreasy and "fixed" the lights with the earlier mentioned deletion of

the files. But indeed this patch seems to be a more permanent fix.

Cannot confirm it though

117 Posts

November 7th, 2017 11:00

Check my reply on this thread.  It should help you fix the pulse if a node gets into that state.

Re: Front pannel LED - Blue (slowly) blinking_OneFS v8.0.0.3

107 Posts

November 8th, 2017 23:00

It seems that there are some improvements made in OneFS 8.0.1.2:

195774

The alert LED on the front panel was lit although there were no unresolved

warnings or critical events. This was a CELOG issue because the LED flag file was

not being cleared as expected. The front panel alert LED behaved as expected.

29 Posts

January 8th, 2018 06:00

We had 3 amber leds on our cluster today.... re-seated the front panels and all blue again...

upgrade scheduled.

No Events found!

Top