Avamar: Troubleshooting HFSCHECK failures due to MSG_ERR_NODE_DOWN

A checkpoint validation (hfscheck) can fail with any number of symptoms. This article addresses the reported error MSG_ERR_NODE_DOWN.

This event can be logged as a Dial Home, an event in the Management Console Server (MCS) UI, or seen on the command line of the Avamar Utility Node:

MCS UI (or dial-home) Event:

<4004> hfscheck of cp.xxxxxxxxxxxxxx failed on error: MSG_ERR_NODE_DOWN

Command line:

status.dpn |grep hfscheck

Last hfscheck failed with result MSG_ERR_NODE_DOWN  

If an Avamar Data Storage Node goes offline during hfscheck, it can cause hfscheck to fail.

If the node was online when hfscheck started, the process attempts to check the integrity of the stripes on the node.

If the node goes offline it cannot check the stripes, hfscheck fails and reports the MSG_ERR_NODE_DOWN message.

1. Verify that all Avamar Data Nodes are online:

Using the command line on the Avamar Utility Node:

a. Log in to the Avamar Utility Node as admin.
b. Run the status.dpn command to verify all nodes are online. For more information about understanding the output of this command, see Avamar: How to Understand the Output Generated by the 'status.dpn' Command.

-- Or -- 

Using the MCS Avamar Administrator UI:

a. Log in to the Avamar Administrator UI
b. Click Server button
c. Click the Server Management tab
d. Verify that all nodes are showing green (OK) status.

2. If a node is offline, contact Dell Technologies Technical Support for further assistance, otherwise, if all nodes are online, go to step 3.

3. Wait for the next scheduled hfscheck to run, and verify that it completes successfully.

To run a hfscheck prior to the next maintenance window, contact Dell Technologies Avamar Technical Support for assistance.

If the next hfscheck does not succeed, contact Dell Technologies Avamar Technical Support for assistance.