Avamar: Metadata Capacity Resolution Path

Metadata Symptoms:

If the Metadata Capacity limit has been reached or exceeded, the Avamar backup scheduler can be suspended and prevent running further scheduled backup jobs. 

Any of the following items indicate a high or full Metadata Capacity:

• The Avamar Administrator UI dashboard may show a Metadata Capacity value equal to or greater than 100%.

• The Management Console Server (MCS) command-line output shows the Metadata Capacity value equal to or greater than 100%:

Example command and output:  

mcserver.sh --status | grep -i metadataUtilization

metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 100.3

• In the Avamar Administrator UI, the scheduler and dispatcher are in a "suspend" state.

• In the mcserver log files, a message is seen similar to "FINE: Suspending dispatcher."

• From a command-line MCS status query, the scheduler is suspended:

mcserver.sh --status | grep suspend

Backup dispatching: suspended

Introduction:

In an Avamar environment with Data Domain integration, backups can be stored on the Data Doman. 

Although the backup data is sent to, and stored on Data Domain, Avamar keeps records of the backups' metadata.
(Metadata, in this environment, is the information about the backup files, file attributes, directories, and so forth from the client operating system).

History:

• As databases only had a few files, there were only a small amount of metadata for Avamar to track.
• Filesystem backups to Data Domain were not supported in Avamar v6.

• This was necessary due to the newly supported Filesystem backups to Data Domain.
  • With filesystem backups, many more different files are backed up, each of whose attributes and Metadata must be tracked by Avamar.
• As metadata is much smaller in size than backup file contents, it can build up and cause other types of resource limits to be reached.
  • Metadata Capacity was added to protect the server from this new type of usage exceeding restrictions and limits.

There are generally one of three situations when Metadata Capacity becomes high or is a concern:

• This situation can be caused from intentional or accidental backups to Avamar instead of Data Domain
• This situation may also have been caused by a previously high Avamar GSAN capacity that occurred prior to integration of Data Domain.

• High Metadata Capacity is falsely reported likely due to a hardware error, or misconfiguration.

1. Log in to the Avamar Utility Node as admin.

2. Run the following command to verify the high or full metadata capacity:

mcserver.sh --status | grep -i metadataUtilization

Examples:

metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 95.95

metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 100.3

3. Review the following article: Avamar: How to gather the information required to troubleshoot capacity issues.

4. Review the following article to verify that there is no other type of Avamar Capacity issues present: Avamar: Capacity Troubleshooting, Issues, and Questions - All Capacity (Resolution Path) 

5. Verify that there is a Data Domain attached by running the following command:

ddrmaint read-ddr-info

Examples:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<avamar dpnid="1234567890" type="default" version="5">
  <datadomain count="1">
    <ddrconfig client-map-default="true" cloud_enabled="true" cloud_unit_name="ECS_Unit1" dd-cert-chain=" -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE-----" ddos-version="7.7.5.25-1078970" ddrcreatetime="1685672811" ddrid="ABCDEF0C18C7E2D75F20E0A1596E66AA2EE764E9" gsan-backup-target-default="true" hostname="dd.compamy.com" index="1" instant-access-limit="32" ipv4-hostname="dd.compamy.com" ipv6-hostname="" max-streams="50" max-streams-for-cp-backup="2" modelno="DD VE" mtree_name="/data/col1/avamar-1643691765" password="Onlh2XP9xEpvI2exqGQwqA==" policy_id="%2Fdata%2Fcol1%2Favamar-1234567890%3AECS_Unit1" serialno="AUDABCDEFGHIJK" token="AQBpud2RFZVF81Vv6NHMK/Y3jecFGhxv0KUif8BYoNk7iQ==" username="<ddboostuser">
      <snmp community="noSejOLJ1lRsg1kEHjWdiA==" version="2">
        <ports getter-setter="161" trap="163">
        </ports>
      </snmp>
      <client-map>
      </client-map>
      <auth-type>credential</auth-type>
    </ddrconfig>
  </datadomain>
</avamar>

<4774>read-ddr-info: MSG_ERR_NOT_PRESENT: ddr_info does not exist. 

If there is no Data Domain attached, this is likely a false positive caused by hardware errors or a misconfiguration. Contact Support for assistance.

If there is a Data Domain attached, and the Metadata Capacity limits have truly been reached or is very high, Contact Support for assistance.
Where possible, include all the information collected and details to help expedite the time to resolution.

In some instances, the Metadata Capacity value can be lowered by tuning parameters. Other situations may require new limitations or hardware changes. The Avamar Support team member will discuss any required actions.

For information about how Metadata Capacity is represented, see Avamar: The metadata utilization value rises but never falls