This KB article is relevant to Avamar systems running versions 7.x +. This article describes the status.dpn command and explains each section of its output.
What is status.dpn command?
Status.dpn command is a perl script that formats a series of avmaint nodelist and Linux OS commands into a user readable format for the purpose of viewing the current status of an Avamar Grid. Similar to other Avamar commands and tools, status.dpn is located in the
/usr/local/avamar/bin/ directory. Status.dpn depends on GSAN being online to run since avmaint interacts with GSAN to run its commands. If GSAN is not running on a data node, no information on that data node, except the node number and offline state, is displayed.
Example of a status.dpn output:
Mon Jan 24 12:00:52 EST 2013 [AVAMAR1] Mon Jan 24 16:00:52 2013 UTC (Initialized Thu Nov 24 10:35:01 2011 UTC)
Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent Full and Stripe Status by Disk
0.0 10.20.30.41 6.1.0-402 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 0.01 15967 1457401 9.1% 9%(onl:403) 9%(onl:388) 9%(onl:387)
0.1 10.20.30.42 6.1.0-402 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.05 15967 1432248 9.1% 9%(onl:389) 9%(onl:395) 9%(onl:394)
0.2 10.20.30.43 6.1.0-402 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.05 15976 1345864 9.1% 9%(onl:377) 9%(onl:399) 9%(onl:401)
0.3 10.20.30.44 6.1.0-402 ONLINE fullaccess mhpu+0hpu+0hpu 0 false 0.00 15969 1608943 9.1% 9%(onl:388) 9%(onl:406) 9%(onl:383)
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable
All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full
Last checkpoint: cp.20121223181633 finished Sun Jan 23 18:17:00 2013 after 00m 27s (OK)
Last GC: finished Sun Jan 23 17:07:26 2013 after 01m 25s >> recovered 342.02 MB (OK)
Last hfscheck: finished Sun Jan 23 18:16:19 2013 after 12m 05s >> checked 2132 of 2132 stripes (OK)
Maintenance windows scheduler capacity profile is active.
The maintenance window is currently running.
Next backup window start time: Mon Jan 24 02:00:00 2013 EST
Next maintenance window start time: Mon Jan 24 10:00:00 2013 EST
The first line of the status.dpn command displays the time and date in the local time zone, hostname of the Avamar Utility node, time and date in UTC, and the time and date when the Avamar grid was initialized (installed) in UTC. UTC is displayed as the data nodes run on UTC time.
Node:
Displays logical node number of each data node. The status.dpn command does not show any status information of the utility node since GSAN does not run on the Utility node.
IP Address:
Displays the IP address of the data node. For Gen4 and Gen4s, this displays the IP address of the internal backup network. All previous hardware generations contain only a single network.
Version:
Displays the version of the GSAN binary on the Avamar data node. The GSAN binary may be appended with a hotfix number if a GSAN hotfix was installed. The version number should match across all Avamar data nodes on the Avamar grid.
State:
Displays the current state code of the Avamar data node.
- ONLINE - GSAN is currently running on this data node. Normal functionality.
- OFFLINE - GSAN is not currently running on this data node. Requires troubleshooting.
Runlevel:
Displays the run level status of the GSAN process per Avamar data node. There are 10 run levels of GSAN.
- Shutdown - Shutting down of the GSAN process.
- Boot - Booting the GSAN process online.
- Connect - GSAN connecting to each data node's GSAN.
- Validate - Validating the stripe DNS table per Avamar data node.
- Startup - Bringing stripes online.
- Sync - Identify which stripes need to be updated, if any, by rebuilding them or using the transaction logs.
- Upgrade - Used when GSAN binary is being updated to a newer version.
- Running - GSAN has started. All stripes are online but not available to accept backups.
- Fullaccess - Grid is available to accept backups.
Srvr+Root+User:
Displays the activity access mode of the Avamar data node.
- Srvr - Avamar GSAN privileges.
- Root - Avamar dpn root user privileges.
- User - All other users privileges
Activity Access Modes
- mhpu+0hpu+0hpu - Full Access - Normal operations.
- mhpu+0hpu+0000 - Admin Mode - When waitcgsan is starting up due to a hfscheck, the GSAN capacity is equal to or greater than the diskreadonly limit and the difference in OS capacity utilization exceeds the freespaceunbalance value listed in avmaint config.
- mhpu+0000+0000 - Sync Mode - During a restart of GSAN while syncing stripes and when testintegrity is running or a stripe is rebuilding.
- m000+0000+0000 - Read Only Mode - During garbage collection and checkpoints. May display as 0000+0000+0000 as well.
Note: In Avamar version 7.x and later, garbage collection is run in Full Access mode and does not put the Avamar grid in Read Only mode
If two activities are running that affect the activity access mode, the most restrictive mode is enabled.
See Appendix #1 for the processes that effect the activity access mode.
Definition of the mhpu bits.
- m=Migrate - Is the system capable of migrating stripes?
- h=HFS Data Store - Is the system capable of writing to the HFS Data Store?
- p=Persistent Store - Is the system capable of writing to the Persistent Store?
- u=User Accounting System - Is the system capable of writing to the User Accounting System?
Dis:
Number of dispatchers in use on the Avamar data node. Dispatcher connections are connections to the GSAN process on the Avamar data node. One dispatcher connection per GSAN connection or GSAN interacting command. Example of a GSAN connection would be a backup session with the avtar process. Example of a GSAN interacting command would be avmaint. See KB article 63548:
How many simultaneous client sessions can be made to the Avamar server? (v6.1 onwards) for information on the number of dispatcher connections that can run per Avamar data node for each software version.
Suspend:
Displays if the dispatchers are suspended or not. If suspended, the Avamar grid prevents new dispatchers being activated for use for backups, restores and replications.
Load:
Instantaneous total load average on the Avamar data node. Similar metrics via the Linux OS "uptime" command.
UsedMB:
Total amount of RAM the Avamar data node is currently using by all processes. Similar metrics via the Linux OS "free -m" command.
Errlen:
In bytes, the length of the /data01/cur/err.log on the Avamar data node.
%Full:
Percent full of the total GSAN capacity of the Avamar data node. Percentage is rounded to a whole number eg. 47%
Percent Full and Stripe Status by Disk:
Displays the number of stripes per partition and their status on the Avamar data node. See Appendix #2 for more information on stripe statuses.
Displays the percent full the partition is according to the GSAN capacity. Percentage is rounded to the first decimal place eg 46.8%
All reported states:
Provides a summary of the status of the grid. Lists the node state, runlevels, and modes. If there are multiple statuses for an item, it lists each one. For example, if a node is offline, we see the following:
All reported states=(ONLINE, OFFLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status:
Displays OK or degraded. Degraded status occurs if a node is in offline state.
Access-Status:
Displays if grid has a restriction in place due to the activity access mode. If a restriction is in place, "admin" is displayed. If no restrictions are in place, "full" is displayed. This section also displays if any stripes are offline or in an OFFLINE_MEDIA_ERROR state.
Maintenance Activities, Tasks, and Schedules:
The first section displays the status of the maintenance activities since GSAN is online. It displays the progress of the maintenance activity and its status of being in progress, completed, or failed with error.
If GSAN was started recently, the maintenance activities' status is cleared. See
Appendix #3 for examples.
The second section displays which schedule is currently in progress, the maintenance tasks currently in progress and the maintenance and backup schedule information.
Definitions of the maintenance tasks:
- script - Runs the GSAN maint scripts at the start of each window. By default, runs for up to 15 minutes uninterrupted, after which it runs in parallel with other tasks until it completes. These scripts are located in /usr/local/avamar/etc/gsan-maint-scripts/ directory.
- indexcacheloader - Loads the index caches at the start of the backup window and the maintenace window if needed, and unloads the index caches at the start of the maintenance window. In 7.x, there be a indexcacheunloader task to show the indexcaches unloading.
- cancel - Cancels/kills backups prior to garbage collection. By default, runs for 15 minutes. Duration is configurable.
- gc/cp/hfscheck - Respective maintenance task is currently running.
- autorepair - Performs auto-repair (if needed) after hfscheck completes.
- conversion - Converts stripes (if needed) at the end of the maintenance window.
- poolcheck - If needed, runs a file pool check at the end of the maintenance window. This runs on Gen3 3.3TB nodes with a file pool.
- balance - (6.1.0 and later) Balances stripes at the end of the maintenance window if there is time available after the maintenance activities have completed. By default, temporarily sets balancemin to 10 (can be configured in the avmaint schedule), unless balancemin has already been set explicitly using avmaint config".
The information for the schedules are set for when Avamar maintenance (garbage collection, checkpoint and hfscheck), and backups are scheduled to start. This section also displays if any of the schedules or individual maintenance activities are stopped or suspended. See
Appendix #4 for the example.