IIRC a drive stall time of 3.5 seconds has become the default a long time ago...

So if your cluster uses a smaller value like 0.5 seconds, increase it as suggested in the KB.

(Make sure to get all the 0s right when expressing 3.5 seconds as 3500000 usec = microseconds )

-- Peter

Hmm I status the node in question and I do see that the threshold is set to 1500000.  I still don't know how this could cause the continuous rebooting issue, unless its the internal boot disk that is stalling.

What is the impact to the user as this setting is being modified?  Will the cluster require a reboot?  Will the change alter the access times getting data on and off the disk?  Can I do this while the cluster is in production during production hours with no impact on users?

There is no impact to changing the drive stall timeout threshold. A drive firmware update would not change the drive stall timeout threshold. Even if you did the upgrade, they would more than likely still recommend making the manual change.

sjones5,

yeah that's the KB I have indicated that support has provided me.  I need to know if there is any impact as stated above and why not go with a disk firmware update over a manual configuration change like this?

Hi chjatwork,

If it helps, the 3.5 second timeout is still well below the industry standard. There is an older article regarding drive stalls that I will link below. While the article is a bit dated, the context still applies. There is also an internal note in that article that states the following:

The 3.5 second stall timeout has been tested both by Isilon Engineering and is used at many customer sites. This configuration change has proved to be safe, and no performance impact to customer workflows has been identified or reported.

OneFS: Introduction to drive stalls

Even if reducing the number of occurring disk stalls might not

finally prevent the node from rebooting,

if will definitely make relevant patterns stand out more clearly,

like coincindent stalls of multiple disks and other anomalies.

And you can go back to 1.5s at any time of course...

As an anecdote: we once did manage to set the stall time to 0.35 seconds accidentally (guess how...)

Which resulted in quite a stall storm, but no serious impact to users during

the 10 minutes or so while it lasted.

And yes, both local and cluster-wide sysctl settings take effect without rebooting.

If support agrees, you can also apply a 'plain' local sysctl change on the affected node,

as transient means for diagnosis. Note that it will revert to the cluster-wide setting after the node boots again.

Peter,

This is good to know.  I may go this route if the 2 nodes in question repeat this same issue.

Thank you,