Avamar: How to understand the output generated by the status.dpn command

This article describes the status.dpn command and explains each section of its output.

What is the status.dpn command?   

The status.dpn command is a perl script that formats a series of "avmaint nodelist" and Linux Operating System (OS) commands into a human-readable format for viewing the current status of an Avamar Grid. 

Similar to other Avamar commands and tools, status.dpn is located in the "/usr/local/avamar/bin/" directory. 

The status.dpn command depends on the GSAN being online to run, as "avmaint" interacts with the GSAN to run its commands.

If the GSAN is not running on a data node, no information about that node, apart from the node number and its offline state, is displayed.

Example of a status.dpn output:

Tue May 13 00:43:37 EDT 2025  [Avamar1.company.com] Tue May 13 04:43:36 2025 UTC (Initialized Fri Aug 26 22:10:38 2022 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0  192.168.255.2  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.03 30672 26994469   0.1%   0%(onl:62 )  0%(onl:44 )  0%(onl:54 )  0%(onl:36 )  0%(onl:38 )  0%(onl:37 )  0%(onl:37 )  0%(onl:42 )  0%(onl:33 )  0%(onl:32 )  0%(onl:30 )  0%(onl:30 )
0.1  192.168.255.3  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   2 false   0.02 30821 26812792   0.1%   0%(onl:40 )  0%(onl:33 )  0%(onl:41 )  0%(onl:33 )  0%(onl:68 )  0%(onl:32 )  0%(onl:33 )  0%(onl:39 )  0%(onl:31 )  0%(onl:36 )  0%(onl:31 )  0%(onl:50 )
0.2  192.168.255.4  19.12.0-186  ONLINE fullaccess mhpu+0hpu+0hpu   0 false   0.02 29008 26927773   0.1%   0%(onl:36 )  0%(onl:39 )  0%(onl:41 )  0%(onl:48 )  0%(onl:40 )  0%(onl:34 )  0%(onl:41 )  0%(onl:32 )  0%(onl:41 )  0%(onl:35 )  0%(onl:34 )  0%(onl:46 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1234567890@EC:2A:72:07:59:23

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20250512120531 finished Mon May 12 08:05:55 2025 after 00m 24s (OK)
Last GC: finished Mon May 12 08:00:26 2025 after 00m 08s >> recovered 0.00 KB (OK)
Last hfscheck: finished Mon May 12 08:04:15 2025 after 02m 54s >> checked 1406 of 1406 stripes (OK)

Maintenance windows scheduler capacity profile is active.
  The backup window is currently running.
  Next backup window start time: Tue May 13 20:00:00 2025 EDT
  Next maintenance window start time: Tue May 13 08:00:00 2025 EDT
 

• The time and date in the local time zone
• The hostname of the Avamar Utility Node
• The time and date in Universal Time Coordinated (UTC)
• The time and date when the Avamar grid was initialized (installed) in UTC. UTC is displayed as the data nodes run on UTC time

• Displays the version of the GSAN binary on the Avamar data node

• • ONLINE

    • The GSAN is running on this data node. Normal functionality
  • OFFLINE
    • The GSAN is not running on this data node. Requires troubleshooting

• Displays the run level status of the GSAN process (per Avamar data node)
• There are multiple run levels of GSAN:

• • Shutdown

    • Shutting down of the GSAN process
  • Boot

    • Booting the GSAN process online
  • Connect

    • The GSAN connecting to each peer data node's GSAN
  • Validate

    • Validating the stripe DNS table on each Avamar data node
  • Startup

    • Bringing stripes online
  • Sync

    • Identifying which stripes must be updated, if any, by rebuilding them or using the transaction logs
  • Upgrade

    • Used when the GSAN binary is being updated to a newer version
  • Running

    • The GSAN has started. All stripes are online but not available to accept backups
  • Fullaccess
    • The grid is available to accept backups

• • Srvr

    • Avamar GSAN privileges
  • Root

    • Avamar dpn root user privileges
  • User
    • All other users privileges

• • mhpu+0hpu+0hpu

    • Full Access
      • Normal operations
  • mhpu+0hpu+0000

    • Admin Mode
      • When "waitcGSAN" is starting up due to a hfscheck, the GSAN capacity is equal to or greater than the "diskreadonly" limit and the difference in OS capacity utilization exceeds the "freespaceunbalance "value listed in "avmaint config"
  • mhpu+0000+0000

    • Sync Mode
      • During a restart of GSAN while syncing stripes and when "testintegrity" is running or a stripe is rebuilding
  • m000+0000+0000
    • Read-Only Mode
      • Displays during checkpoints

(May also display as 0000+0000+0000)

See Appendix #1 for the processes that effect the activity access mode.

• • m=Migrate

    • Is the system capable of migrating stripes?
  • h=HFS Data Store

    • Is the system capable of writing to the HFS Data Store?
  • p=Persistent Store

    • Is the system capable of writing to the Persistent Store?
  • u=User Accounting System
    • Is the system capable of writing to the User Accounting System?

• Displays the dispatcher status (suspended or not)
• If suspended, the Avamar grid prevents new dispatchers being activated for use for backups, restores, and replications

• Total amount of RAM the Avamar data node is currently using by all processes

• A summary of the status of the grid
• Lists the node state, runlevels, and modes
• If there are multiple statuses for an item, it lists each of them.

All reported states=(ONLINE, OFFLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu) 

• The first section displays the status of the maintenance activities since GSAN is online.
• It displays the progress of the maintenance activities and their status (In progress, completed, or failed with error)
• If the GSAN was started recently, the maintenance activities' status is cleared

See Appendix #3a and #3b for examples.

• The second section displays which schedule is currently active, any maintenance tasks in progress, and the maintenance and backup schedule information

• • Script 

    • Runs the GSAN maintenance scripts at the start of each window. By default, it runs for up to 15 minutes uninterrupted, after which it runs in parallel with other tasks until it completes.
    • These scripts are located in the "/usr/local/avamar/etc/GSAN-maint-scripts/" directory
  • indexcacheloader

    • Loads the index caches at the start of the backup window and the maintenance window if needed, and unloads the index caches at the start of the maintenance window.
    • From v 7.x, there is an  "indexcacheunloader" task to show the "indexcaches"  unloading
  • Cancel

    • Cancels or kills backups prior to garbage collection
  • gc/cp/hfscheck

    • Respective maintenance task is currently running
  • autorepair

    • Performs autorepair (if needed) after hfscheck completes
  • Conversion

    • Converts stripes (if needed) at the end of the maintenance window
  • Balance
    • Balances stripes at the end of the maintenance window if there is time available after the maintenance activities have been completed.
    • By default, temporarily sets "balancemin" to 10 (can be configured in the avmaint schedule), unless "balancemin" has already been set explicitly using "avmaint config"

• The information for the schedules is set for when Avamar maintenance (garbage collection, checkpoint and hfscheck), and backups are scheduled to start

• This section also displays if any of the schedules or individual maintenance activities are stopped or suspended.

See Appendix #4 for the example.

Appendix #1 - Activity access modes:

The following are items that can change the access mode of the Avamar grid.

Run the following command to determine what process has changed the activity access mode of the Avamar grid.

avmaint nodelist --xmlperline=99 | grep activityaccessmodes 

Sample output:

<activityaccessmodes adminuser="mhpu+0hpu+0hpu" checkpoint="mhpu+0hpu+0hpu" conversion="mhpu+0hpu+0hpu" diskbeat="mhpu+0hpu+0hpu" garbagecollect="mhpu+0hpu+0hpu" heartbeat="mhpu+0hpu+0hpu" hfscheckserver="mhpu+0hpu+0hpu" hfscheckexecute="mhpu+0hpu+0hpu" nodebeat="mhpu+0hpu+0hpu" runlevel="mhpu+0hpu+0hpu" testintegrity="mhpu+0hpu+0hpu" removehashes="mhpu+0hpu+0hpu" rebuildstripe="mhpu+0hpu+0hpu" diskfull="mhpu+0hpu+0hpu"/>

Appendix #2 -  Stripe Status:

• ONL - Online - (onl:2961)
  • Stripe is online without issue
  • Normal stripe status
• OFF - Offline - (onl:2960,OFF:1)
  • Stripe is offline
  • This could be due to a stripe being rebuilt, or a stripe is put offline due to an error
• ERR - Error - (onl:2959,ERR: 2)
  • Stripe is in the OFFLINE_MEDIA_ERROR state
  • See Avamar: Troubleshooting Stripes Offline with OFFLINE_MEDIA_ERROR Errors (Resolution Path) for more information
• SUS - Suspended - (onl:1,SUS:2960)
  • Stripe is likely suspended due to disk I/O performance and has caused Avamar to suspend the data partition
  • See Avamar: Suspended Partitions, Stripes, and Hfscheck Failures on Avamar for more information
• MIG - Migrating - (onl:2960,MIG:1) 
  • Stripe is moving or migrating to a different Avamar data node
• RES - Restarting - (onl:2040,RES:920)
  • Stripe is restarting from its previous state of offline

Appendix #3a - Sample output for maintenance activities that have run on the Avamar grid:

Last checkpoint: cp.20121223181633 finished Sun Jan 23 18:17:00 2013 after 00m 27s (OK)
Last GC: finished Sun Jan 23 17:07:26 2013 after 01m 25s >> recovered 342.02 MB (OK)
Last hfscheck: finished Sun Jan 23 18:16:19 2013 after 12m 05s >> checked 2132 of 2132 stripes (OK)

Appendix #3b - Sample output of the maintenance activities after a GSAN restart:

No checkpoint yet
No GC yet
No hfscheck yet 

Appendix #4 - Sample output of the maintenance schedule, checkpoint, garbage collection, and hfscheck being suspended:    

Maintenance windows scheduler capacity profile is active.
  WARNING: cp is suspended temporarily.
  WARNING: gc is suspended temporarily.
  WARNING: hfscheck is suspended temporarily.
  The backup window is currently running 
  Next backup window start time: Tue May 13 21:00:00 2025 PDT
  Next maintenance window start time: Tue May 13 14:00:00 2025 PDT

Appendix #5 - status.dpncommand options:

• status.dpn <interval in seconds>
  • Runs the status.dpn command in a loop for the number of seconds specified
    • A value of 10 (Ten) returns the status.dpn output every 10 s until the command is manually exited
    • A value of 0 (zero) returns a single status.dpn output without looping
• --help
  • Displays the status.dpn help menu
• --short
  • Display fewer details in status.dpn
    • Removes the hostname, version, load, UsedMB %Full and Percent Full and Stripe Status by Disk information
  • Used for faster displaying status.dpn outputs and troubleshooting purposes
• --sort
  • Sorts the nodes by a different variable option:
    • =node
      • Sorts the data nodes by logical node numbers (Default)
    • =ipaddr
      • Sorts the data nodes by IP address
    • =dispatcher
      • Sorts the data nodes by the number of dispatchers in use (Sorted with the lowest number of dispatchers in use at the top of the list)
    • =load
      • Sorts the data nodes by the load value (Sorted with the lowest load value at the top of the list)
    • =used
      • Sorts the data nodes by the UsedMB value (Sorted with the lowest value of UsedMB are at the top of the list)
    • =full
      • Sorts the data nodes by the %Full value (Sorted with the lowest value of %Full are at the top of the list)