Avamar: Understanding Encryption-at-rest functionality
Summary: This article discusses aspects of Avamar's encryption-at-rest functionality including its purpose and caveats. The internal sections of this article, which is available to Dell staff and partners only, explain how to configure it. ...
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Instructions
The purpose of this article is to discuss the use of encryption-at-rest on an Avamar server.
This is an advanced feature and assumes that you are experienced with regular Avamar commands used for system maintenance and support activities. Otherwise, do not proceed. Consult an appropriately skilled person.
The Avamar Technical Addendum documents the "avmaint atrestencryption" command but does not describe how to safely implement it on a production Avamar system.
This article describes the implementation and answers some common questions relating to encryption-at-rest.
These deduplicated ('atomic') data chunks are written to .dat files. Metadata and index data are written to other stripe files (.inx, .chd, .cdt and so on).
This process 'shreds' the backup data since it is broken up into small pieces and scattered around the system.
Although entire files cannot be read, it is possible to view fragments of files. If we run 'strings' on a data stripe, pieces of plain text data may be visible.
Below is an example from a single node Avamar:
1. What problem does encryption-at-rest solve?
Encryption-at-rest encrypts the data stripes which contain the user backup data.
Once enabled, if somebody stole Avamar or its disk drives, they would need to decrypt the data before it could be read.
Encryption-at-rest does prevent someone with Avamar application or OS level access from logging into the Avamar system and restoring data. This is possible using the UI or by running avtar commands and restoring files within the GSAN to the utility node.
2. When can encryption-at-rest be enabled?
Encryption-at-rest can be enabled during the installation workflow so that it is in effect from the beginning of the Avamar server's life.
During this process, the installer prompts for a salt and a password for encryption-at-rest. If a salt and a password are specified, the feature is enabled.
After a system has been initialized enable encryption-at-rest can be enabled. However, running the avmaint command after initialization affects only newly created data stripes and data stripes that are later crunched.
Although we can be confident that crunching eventually occurs on every unencrypted stripe, we cannot guarantee it. Stripes created before encryption-at-rest is enabled may remain unencrypted for a long time. Sometimes, possibly forever.
For example, there is a possibility that Avamar could fill up a stripe with static data (such as MCS flushes which contain sensitive information like hostnames). If that stripe never changes, it never undergoes garbage collection or crunching and will never get encrypted.
Therefore, only enable encryption-at-rest as part of the installation workflow.
3. Performance overhead of using encryption-at-rest
A common question is "Why must the diskreadonly limit be lowered on systems set with encryption-at-rest?"
Encrypting and decrypting all the stripes stored in an Avamar GSAN requires a great deal of additional work to be performed by storage nodes.
Engineering have observed that the additional processing overhead required is up to 50% more than a system where encryption-at-rest is not enabled.
Therefore, on Avamar where encryption-at-rest is enabled, diskreadonly limit MUST be reduced from the default 65% down to 40%. This means that 1.5x the number of nodes are required to store the same amount of data as one whose stripes are not encrypted.
This is in order to ensure that acceptable performance, and reliability are maintained.
4. Can encryption-at-rest be disabled?
Once encryption-at-rest has been enabled on an Avamar system, it cannot be disabled, only the encryption salts can be changed.
5. How to demonstrate that encryption-at-rest is enabled.
On an Avamar system where encryption-at-rest has been enabled you should see output similar to the following
avmaint nodelist --xmlperline=9999 | grep atrestencryption
Information similar to the following appears in the command shell:
<atrestencryption-status enabled="true" nr-salts="1"/>
Where:
enabled="true" nr-salts="1" indicates that at-rest encryption is enabled.
enabled="false" nr-salts="0" indicates that at-rest encryption is not enabled.
Example:-
This is an advanced feature and assumes that you are experienced with regular Avamar commands used for system maintenance and support activities. Otherwise, do not proceed. Consult an appropriately skilled person.
The Avamar Technical Addendum documents the "avmaint atrestencryption" command but does not describe how to safely implement it on a production Avamar system.
This article describes the implementation and answers some common questions relating to encryption-at-rest.
- What problem does encryption-at-rest solve?
- When can encryption-at-rest be enabled?
- Performance overhead of using encryption-at-rest
- Can encryption-at-rest be disabled?
- Procedure to enable encryption-at-rest (available to Dell internal staff and partners only)
- How to demonstrate that encryption-at-rest is enabled
These deduplicated ('atomic') data chunks are written to .dat files. Metadata and index data are written to other stripe files (.inx, .chd, .cdt and so on).
This process 'shreds' the backup data since it is broken up into small pieces and scattered around the system.
Although entire files cannot be read, it is possible to view fragments of files. If we run 'strings' on a data stripe, pieces of plain text data may be visible.
Below is an example from a single node Avamar:
admin@utility:/data01/cur/>: strings 000000000000002E.dat
5900>: Workorder received:
<restore pid="
firm="true" ack
msgver="5"
timeout="10"
="XXXD
ath="Mailbox Database 0904743075" key
sync="fg" cd
d15696dae634d00c120b45e56d0ad2410380d945" mx
6064/776
0/33" wr
<targets>.D
tem_info/BL)
upComponentsDocument.xml
{76fe1ac4-15f7-4bcd-987e-8t
b462fb7}01
_chain
Since the data chunks may contain email addresses, IP addresses, hostnames, or other sensitive information, some consider this to be an unacceptable security risk.
1. What problem does encryption-at-rest solve?
Encryption-at-rest encrypts the data stripes which contain the user backup data.
Once enabled, if somebody stole Avamar or its disk drives, they would need to decrypt the data before it could be read.
Encryption-at-rest does prevent someone with Avamar application or OS level access from logging into the Avamar system and restoring data. This is possible using the UI or by running avtar commands and restoring files within the GSAN to the utility node.
2. When can encryption-at-rest be enabled?
Encryption-at-rest can be enabled during the installation workflow so that it is in effect from the beginning of the Avamar server's life.
During this process, the installer prompts for a salt and a password for encryption-at-rest. If a salt and a password are specified, the feature is enabled.
After a system has been initialized enable encryption-at-rest can be enabled. However, running the avmaint command after initialization affects only newly created data stripes and data stripes that are later crunched.
Although we can be confident that crunching eventually occurs on every unencrypted stripe, we cannot guarantee it. Stripes created before encryption-at-rest is enabled may remain unencrypted for a long time. Sometimes, possibly forever.
For example, there is a possibility that Avamar could fill up a stripe with static data (such as MCS flushes which contain sensitive information like hostnames). If that stripe never changes, it never undergoes garbage collection or crunching and will never get encrypted.
Therefore, only enable encryption-at-rest as part of the installation workflow.
3. Performance overhead of using encryption-at-rest
A common question is "Why must the diskreadonly limit be lowered on systems set with encryption-at-rest?"
Encrypting and decrypting all the stripes stored in an Avamar GSAN requires a great deal of additional work to be performed by storage nodes.
Engineering have observed that the additional processing overhead required is up to 50% more than a system where encryption-at-rest is not enabled.
Therefore, on Avamar where encryption-at-rest is enabled, diskreadonly limit MUST be reduced from the default 65% down to 40%. This means that 1.5x the number of nodes are required to store the same amount of data as one whose stripes are not encrypted.
This is in order to ensure that acceptable performance, and reliability are maintained.
4. Can encryption-at-rest be disabled?
Once encryption-at-rest has been enabled on an Avamar system, it cannot be disabled, only the encryption salts can be changed.
5. How to demonstrate that encryption-at-rest is enabled.
On an Avamar system where encryption-at-rest has been enabled you should see output similar to the following
avmaint nodelist --xmlperline=9999 | grep atrestencryption
Information similar to the following appears in the command shell:
<atrestencryption-status enabled="true" nr-salts="1"/>
Where:
enabled="true" nr-salts="1" indicates that at-rest encryption is enabled.
enabled="false" nr-salts="0" indicates that at-rest encryption is not enabled.
Example:-
avmaint nodelist --xmlperline=9999 | grep atrestencryption
<atrestencryption-status enabled="true" nr-salts="1"/>
<atrestencryption-status enabled="true" nr-salts="1"/>
<atrestencryption-status enabled="true" nr-salts="1"/>
<atrestencryption-status enabled="true" nr-salts="1"/>
On a system where encryption-at-rest is performed at initialization or before any stripes are created, once stripes are created and encrypted they contain only gibberish if we try to read them using the 'strings' command as previously.
admin@utility:/data01/cur/>: strings 0000000000000656.dat | less
p T7T5s| 1 N © M1 c]LP ,iCE ol v < #6U, ~ )XW D TU ( A: ) ~ ,t} '
3ET S _b kE l>5Q T yc ®x` `' s Kg 4g sh . 8 c! e , A M a M} kX; A 1v z
_ { !{a X JUS * nD 8+ TE2Tf S h,37 QX G * " ®u +8 ' !{+2w 6 PE
Additional Information
Frequently asked questions:
How should Griffin systems (Avamar/DataDomain combination) be configured with the feature?
Both Avamar and Data Domain should be configured with encryption-at-rest separately.
Dell EMC Data Domain Encryption - Frequently Asked Questions
Is the 40% diskreadonly level still applicable to the latest Avamar Datastore hardware?
Engineering were consulted about this in September 2016. The 40% diskreadonly setting applies to the latest hardware (Gen4T). Product management is considering the request but have no plans to reevaluate the limit.
If a customer has a replicating pair of Avamar servers, must both systems be configured with E.A.R?
No, each system is independent. When data on an encrypted system is replicated to an unencrypted Avamar, the target system creates unencrypted stripes.
Similarly, if an unencrypted Avamar replicates data to an encrypted system, the backups are written to encrypted stripes.
Once enabled, can encryption-at-rest be disabled?
No. To achieve this, the data must be replicated to an Avamar server where encrypt-at-rest is not set.
If a customer has a replicating pair of Avamar servers, must both systems be configured with E.A.R?
No, each system is independent. When data on an encrypted system is replicated to an unencrypted Avamar, the target system creates unencrypted stripes.
Similarly, if an unencrypted Avamar replicates data to an encrypted system, the backups are written to encrypted stripes.
Once enabled, can encryption-at-rest be disabled?
No. To achieve this, the data must be replicated to an Avamar server where encrypt-at-rest is not set.
Affected Products
AvamarProducts
Avamar, Avamar ServerArticle Properties
Article Number: 000046249
Article Type: How To
Last Modified: 10 Feb 2025
Version: 5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.