Avamar: Troubleshooting HFSCHECK failures due to MSG_ERR_SHUTDOWN

Summary: This article explains why a checkpoint validation (hfscheck) can fail with the error MSG_ERR_SHUTDOWN.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

A checkpoint validation (hfscheck) can fail for any number of reasons with many symptoms.

This article addresses the hfscheck error MSG_ERR_SHUTDOWN.

This event can be logged as a Dial Home event, a UI event or seen on a putty session on the Avamar Utility Node.

 

UI Event: 

<4004> hfscheck of cp.xxxxxxxxxxxxxx failed on error: MSG_ERR_SHUTDOWN
 

Output from status.dpn: 

Last hfscheck failed with result MSG_ERR_SHUTDOWN

Cause

The Avamar Server processes (gsan) were shutting down when hfscheck was started.

Resolution

Do not attempt to start hfscheck when the Avamar server is being shut down.

1. Verify if other maintenance tasks are (or were) running when hfscheck is being started.

This can be done using the MC UI, the AUI, or using the command line on the Avamar Utility Node.

 

To check using the command line:

a. Log in to the Avamar server as admin and load the admin keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys. 

b. Using the status.dpn command, verify the state of the grid and when the maintenance tasks last ran:

status.dpn 
Wed Apr 23 19:31:35 EDT 2025  [avamar.company.com] Wed Apr 23 23:31:35 2025 UTC (Initialized Fri Aug 26 22:10:38 2022 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0  192.168.255.2  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.56 25016 26430018   0.1%   0%(onl:62 )  0%(onl:44 )  0%(onl:54 )  0%(onl:36 )  0%(onl:38 )  0%(onl:37 )  0%(onl:37 )  0%(onl:42 )  0%(onl:33 )  0%(onl:32 )  0%(onl:30 )  0%(onl:30 )
0.1  192.168.255.3  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   2 false   0.54 25669 26217119   0.1%   0%(onl:40 )  0%(onl:33 )  0%(onl:41 )  0%(onl:33 )  0%(onl:68 )  0%(onl:32 )  0%(onl:33 )  0%(onl:39 )  0%(onl:31 )  0%(onl:36 )  0%(onl:31 )  0%(onl:50 )
0.2  192.168.255.4  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   0 false   0.50 24131 26343301   0.1%   0%(onl:36 )  0%(onl:39 )  0%(onl:41 )  0%(onl:48 )  0%(onl:40 )  0%(onl:34 )  0%(onl:41 )  0%(onl:32 )  0%(onl:41 )  0%(onl:35 )  0%(onl:34 )  0%(onl:46 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1234567890@EC:2A:72:07:59:23

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20250423120430 finished Wed Apr 23 08:04:54 2025 after 00m 24s (OK)
Last GC: finished Wed Apr 23 08:00:26 2025 after 00m 07s >> recovered 0.00 KB (OK)
Last hfscheck: finished Wed Apr 23 08:04:13 2025 after 02m 51s >> checked 1406 of 1406 stripes (OK)

Maintenance windows scheduler capacity profile is active.
  The maintenance window is currently running.
  Next backup window start time: Wed Apr 23 20:00:00 2025 EDT
  Next maintenance window start time: Thu Apr 24 08:00:00 2025 EDT
 
Note: For additional assistance understanding this command see Avamar: How to Understand the Output Generated by the 'status.dpn' Command
 
 

2. If the problem persists contact Dell Technologies Technical Support for additional assistance.

Affected Products

Avamar, Avamar Server
Article Properties
Article Number: 000169742
Article Type: Solution
Last Modified: 27 Apr 2025
Version:  5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.