Dell EMC Data Domain Encryption - Frequently Asked Questions

Encryption Configuration

Question: How is Encryption (DARE) Configured in the DD?
Answer: DARE Encryption can be configured in the DD by the following steps:

1. Add Encryption License

2. Add Security Officer and enable Security Officer Authorizations

3. Enable Encryption 

1) Add Encryption License:
Have a license file with a valid Encryption license added to it.
Use the below command to update the elicense in the DD using the license file available.

elicense update

2) Add Security Officer and Enable SO Authorizations:
a) Add a user with a role as "security" using the command: 

user add secuser role security

b) Enable Security Officer Authorization in the setup using the command: 

authorization policy set security-officer enabled

3. Enable DARE Encryption:
Enable DARE Encryption using the command:

filesys encryption enable

Question: Which platforms are supported with the data-at-rest encryption feature?
Answer: The data-at-rest encryption feature is supported on all Data Domain systems, except on Encryption Disablement Project (EDP) systems.

Question: How can the user store their data in clear text in DD?
Answer: User can ensure that the data is saved in clear text and not encrypted in the DD, by confirming that Encryption is turned off in the setup.
Users can disable encryption in DD using the command: 

filesys encryption disable

Question: Which backup applications/Protocols are supported with the data-at-rest encryption feature?
Answer: DD DARE feature is independent of the underlying backup application or the protocol used by the DD.

Question: Which encryption algorithms can be selected on the Data Domain system?
Answer: The DD Encryption software supports AES 128 or 256-bit algorithm using Cipher Block Chaining (CBC) or Galois Counter Mode (GCM). 

GCM is a mode of operation for symmetric key cryptographic block ciphers. It is an authenticated encryption algorithm designed to provide both authentication and privacy (confidentiality). As the name suggests, GCM combines the well-known counter mode of encryption with the new Galois mode of authentication. The authentication aspect of GCM guarantees that the data that was encrypted was done so by the Data Domain system and was not "injected" by some other means. This differs from CBC where data is encrypted (privacy aspect), but there is no check for the authenticity of the encrypted data.

In CBC mode, each block of plain text is exclusive ORed with the previous cipher text block before being encrypted. This way, each cipher text block depends on all plain text blocks processed up to that point. Also, to make each message unique, an initialization vector must be used in the first block. CBC only guarantees the privacy (confidentiality) of the data through encryption. No authentication of the encryption algorithm or process is done.

Question: How can we change the encryption algorithm in the DD?

Answer: Use the below command if you want to set a specific encryption algorithm in the setup.

filesys encryption algorithm set

Example:

# filesys encryption algorithm set {aes_128_cbc | aes_256_cbc | aes_128_gcm | aes_256_gcm}

Question: How do we ensure that encryption is done on preexisting data once encryption is enabled?
Answer: We can force DDFS to encrypt the preexisting data using the below command:

# filesys encryption apply-changes

This makes the next clean cycle considerably longer and more resource intensive than normal.

Question: How do we disable encryption in the DD?
Answer: Disable the encryption feature in the DD using the encryption disable command: 

filesys encryption disable

This only disables encryption for incoming I/O. Existing encrypted data remains encrypted until it is manually decrypted using apply-changes.

Question: Which encryption commands require DD file system restart in order to take effect?
Answer: The following encryption commands require a DD file system restart in order to take effect:

filesys encryption enable/disable - Enables/disables encryption on the Data Domain system.

filesys encryption algorithm set - Allow the user to select a cryptographic algorithm.

filesys encryption algorithm reset - Resets the encryption algorithm to AES 256 in CBC mode (the default).

​​​
Question: Which encryption commands require the Data Domain file system to be disabled in order to set/use them?
Answer: The following encryption commands require the Data Domain file system to be disabled in order to set or use them:

encryption passphrase change

encryption lock/unlock

​​​​​General Encryption Questions

Question: Is DD Encryption supported on all DD systems?
Answer: The DD Encryption software option is supported on DD systems if it is not an encryption disablement project (EDP) which are the systems that do not allow encryption to be enabled, mainly sold in the Russia region systems.

Question: How is the cryptography performed in DD systems?
Answer: Cryptography is done using OpenSSL and RSA BSafe (RSA BSafe is FIPS 140-2 validated cryptography library used by DD systems to encrypt data at rest) libraries in DDOS. 

Question: What version of BSafe is used with the system?
Answer: As of DDOS 7.10, used BSafe versions are "BSAFE Micro Edition Suite 4.4.0.0" and "BSAFE Crypto-C Micro Edition: 4.1.4.0"

Question: What are the available user interfaces to configure encryption in DDOS?
Answer: Encryption can be configured using CLI, UI or by using REST APIs. REST API supported added in DDOS release 8.0 onwards.

Question: Selective encryption of data is possible? Like only one MTree or file?
Answer: Selective encryption is NOT possible. Encryption can only be enabled or disabled systemwide and not selectively. For systems with cloud support, encryption can be enabled or disable at the cloud-tier and cloud unit granularity.  

Question: Are any cryptographic keys or account passwords transmitted or stored in clear text or under weak ciphers, such as when an entity authenticates, in datafile, in programs, or in authentication directories?
Answer: Absolutely Not.

Question: What version of OpenSSL is used with the system?
Answer: As of DDOS 7.10 release, the openssl version is "OpenSSL 1.0.2zd-fips"

Question: How does data-at-rest encryption protect against data access from users and applications?
Answer: 

• Encryption of data at rest is all about encrypting the data, which resides on the disk subsystem. Encryption or decryption happens at the compression layer. Users or applications send and receive clear text data to the Data Domain system, but any data physically residing on the Data Domain system is encrypted.
• All encryption occurs below the file system and namespace and is invisible to the users or applications. If a user or application already has authorized access to a file or directory, the data can be read in its native format regardless of encryption.
• DD Encryption is designed such that if an intruder circumvents other network security controls and gains access to encrypted data, without the proper cryptographic keys, the data is unreadable and unusable to that person. 

Question: Will encryption happen after the deduplication?
Answer: Yes, encryption happens on deduplicated data. Data is encrypted before being stored on the disk. 

Question: How does the Data Domain ensure the security of the data?
Answer: Data is secured using the encryption of data at rest feature. In addition, when the device is removed (head-swap, file system lock) the passphrase is removed from the system. This passphrase is used to encrypt encryption-keys, so the data is further protected.

Question: What alerts are generated with encryption?
Answer: Alerts are generated in the following cases:

• When there are compromised encryption keys are present
• When the encryption key table is full and no more keys can be added in the system
• When auto key exporting fails
• When automated key rotation fails
• When encryption is disabled
• When the system passphrase changed

Question: Is there any security certification for DDOS? 
Answer: Data domain systems meet with FIPS 140-2 compliance. 

Question: Where does the encryption key get stored?
Answer: Encryption keys are stored persistently in a collection partition in the DDOS system.

Question: If someone pulls out a hard drive from a Data Domain system, can you decrypt data from it?
Answer: Encryption keys are encrypted using system passphrase which is stored in the system head. Even though encryption keys are stored in the disk, without system passphrase, encryption keys cannot be decrypted. So without knowing the key which used to encrypt the data, decryption is not possible from a hard drive.  

Question: What cryptographic keys and passwords are needed for recovery, especially for disaster recovery?
Answer: Keys can be exported in a secure file and kept external to the system. Recovery of this file is done with the help of engineering. Also at the time of recovery, the customer must know the passphrase that they used at the time of keys export command.

Question: How to lock file system prior to transit of system to remote location.
Answer: Below is the procedure for it: 

• Disable the file system:

  sysadmin@itrdc-DD630-42# filesys disable

• Lock the file system and enter a new passphrase (this requires authentication by the above security user):

  sysadmin@itrdc-DD630-42# filesys encryption lock
  This command requires authorization by a user having a 'security' role.
  Please present credentials for such a user below.
          Username: secuser
          Password:
  Enter the current passphrase:
  Enter new passphrase:
  Re-enter new passphrase:
  Passphrases matched.
  The filesystem is now locked.

• The new passphrase must NOT be lost or forgotten. Without this passphrase, the file system cannot be unlocked meaning that data on the DDR cannot be accessed. To unlock the system when it reaches a remote location use below command:

sysadmin@itrdc-DD630-42# filesys encryption unlock
This command requires authorization by a user having a 'security' role.
Please present credentials for such a user below.
        Username: secuser
        Password:
Enter the passphrase:
The passphrase has been verified. Use 'filesys enable' to start the filesystem.

• The file system can now be enabled and used as normal

Question: Does the storage sanitize command have any relationship with file system encryption?
Answer: No, file system encryption and storage sanitizing are two independent features. 

Question: Is over the wire encryption supported in the EDP (encryption disablement project) system?
Answer: We cannot enable Data at Rest Encryption (DARE) or over the wire encryption (with replication or with ddboost) in EDP system.

System Passphrase 

Question: What is the system passphrase?
Answer: DDOS has the provision to secure credentials within the system by setting a system level passphrase. The passphrase is a human readable (understandable) key, like a smart card, which is used to generate a machine-usable AES 256 encryption key. 

It provides two benefits:

• It allows the administrator to change the passphrase without having to manipulate the encryption keys. Changing the passphrase indirectly changes the encryption of the keys, but does not affect user data. Changing the passphrase does not change the underlying Data Domain system encryption key. It changes the Encryption of the Data Domain system key, but the system key stays the same.
• It allows a physical Data Domain system to be shipped with an encryption key on the system but without the passphrase being stored on it. This way if the box is stolen in transit, an attacker cannot easily recover the data since the system only has encrypted keys and encrypted data.

The passphrase is stored internally on a hidden part of the Data Domain storage system. This allows for the Data Domain system to boot and continue servicing data access without administrator intervention.

Creating or changing the passphrase:

• The system passphrase can be created using the CLI after an administrator authenticates with the Data Domain system.
• The system passphrase can be changed using the CLI after an administrator and a security role user (like a security officer) authenticate with the Data Domain system (such that no single administrator can make changes independently.)

Question: When is the passphrase used?
Answer: The system passphrase is used as a primary key by various DDOS components which include file system encryption, cloud access, certificate management, Boost tokens, system configuration modules in scale-out environments, and licensing information. DDOS software provides mechanisms to set and modify this system passphrase. It also provides options to control whether the system passphrase is stored on disk which is especially used for enhanced security when the DD system is being transported. 

Question: How is the passphrase used for secure transport of the DD system?
Answer: The process uses the "filesys encryption lock" command, which allows the user to lock the file system by changing the passphrase. The user enters a new passphrase that re-encrypts the encryption key, but the new passphrase is not stored. The encryption keys are not recoverable until the file system is unlocked using the "filesys encryption unlock" command.

The process is described in the Confluence Lab Security Configuration Guide.

Question: What happens if the passphrase changes? Can data still be accessed?
Answer: Yes, changing the passphrase does not change the underlying Data Domain system encryption key, only the encryption of the encryption key. Therefore, data access is not impacted.

Question: How can we query if a passphrase is set on the system?
Answer: If a passphrase is set on the system, running the "system passphrase set" command throws an error indicating that the passphrase is already set.

Question: What happens if the passphrase is lost or forgotten?
Answer: If the customer loses the passphrase while the box is locked, they lose their data. There is no back-door or alternative way of getting access to it. Without a good process for managing that passphrase, this could happen accidentally and they would not be able to recover the key or data. However, the encrypted key can never be lost or corrupted because of the system’s integrated protection mechanisms.

Question: Is there any mechanism to reset the forgotten system passphrase?
Answer: System passphrase can be reset by force only in certain scenarios with the help of Customer Support. The force-update mechanism introduced in DDOS 7.2 can be used for this only if specific conditions are met. More details are in KB article 20983, Data Domain: How to Reset a lost system passphrase in DDOS v7.2 or later (log in to Dell Support required to view article)

Question: Is there an option to avoid storing system passphrase on the DD system?
Answer: The system passphrase by default is stored in a hidden location on the Data Domain system. The system option "system passphrase option store-on-disk" can be used to change this and avoid storing passphrase on-disk.

Embedded Key Manager (EKM)

Top-level command:

system# filesys encryption embedded-key-manager <option>

Question: Is key rotation supported with Embedded Key Manager?
Answer:  Yes, key rotation per Data Domain system is supported with Embedded Key Manager. Through the UI or CLI the administrator can set up a key rotation period (weekly or monthly).

Question: Is there a charge for the embedded key management functionality?
Answer: There is no charge for this functionality. This functionality is included as part of the standard DD Encryption software license option.

Question: Can the customer move from local key management to external key management (EKM)?
Answer

• Yes, external key managers can be enabled at any time. However, the local keys being used remain on the Data Domain system. External key managers are not able to manage the EKM keys. Existing data does not require re-encrypting. If for a customer, compliance data must be re-encrypted with EKM keys, then it must be done manually using apply-changes with the new RW key. Destroying EKM keys after a switch is not mandatory.
  • key-manager switch automatically switches the active key to the key from KMIP. 
  • Example of how the KMIP key MUID looks when a switch happens

    Key-ID     Key MUID                                                                    State                     Key Manger Type
    1               be1                                                                    Deactivated               DataDomain
    
    2               49664EE855DF71CB7DC08309414C2B4C76ECB112C8D10368C37966E4E2E38A68       Activated-RW              KeySecure

Question: What happens when we have key rotation disabled or enabled?
Answer: 

• Key rotation disabled is the default encryption functionality if you do not enable key rotation from the UI or CLI. In that scenario all the data are encrypted with the existing active key.
• If key rotation is enabled, based on the rotation frequency we rotate the keys, and the data is encrypted with the latest active key.

External Key Managers

Question: What are the external key managers supported on DD?
Answer: We are supporting the below external key managers:

• Gemalto KeySecure (support added in DDOS release 7.2)
• Vormetric (support added in DDOS release 7.3)
• CipherTrust (support added in DDOS release 7.7)
• IBM GKLM (support added in DDOS release 7.9)

Question: Is there a separate license required to enable the integration with an external key manager?
Answer: Yes, a separate license is needed from the respective vendor to integrate External Key Manger with DD.

Question: How many key managers support at a time?
Answer: Only one key-manager can be active at any given point in time on a DD system.

Question: Where do I find more information about how to configure KMIP External Key Manager?
Answer: The KMIP integration Guide for DDOS has detailed information for configuring the different external key managers supported by DD.

Question: How are certificates managed for External Key Managers in DD?
Answer: While configuring the External key manager, we have to generate a CA certificate (CA certificate could be a self-signed certificate or signed by third party) and host certificate. Once configuration is done on the External Key manager server, users have to import the CA certificate and Host certificate on the DD system. Then we configure and enable the external key manager. 

Question: What is CA?
Answer: A Certificate Authority (CA) acts as the initially trusted shared entity between peers and issues signed certificates to make it possible for each party to trust the other. A certificate generally acts as the identity of a server or client. 

Question: What is a local CA signed certificate, what is a CA signed certificate?
Answer: A CA signed certificate is a certificate that has been issued and signed by a publicly trusted certificate authority (CA). A CA signed certificate is trusted automatically. A local CA can issue signed certificates since the private signing key is stored inside the Key Manager system. An external CA does not store the private key. Instead an external CA is used as a trusted entity for various interfaces and services inside the system.

Question: How to create a certificate signing request in DD?
Answer: On Data Domain System, generate the CSR, using the below CLI command. Through this way, the private key is never exposed to the external Key manager.

adminaccess certificate cert-signing-request

Question: Is it possible to switch between Key Managers?
Answer: Switching between External Key Manger to Embedded Key Manager is allowed and is seamless. But switching from Embedded Key Manager to External Key Managers requires appropriate certificate installation and configuration. Switching between two External Keys Managers (for example: KMIP-CipherTrust, DSM-Ciphertrust, CipherTrust to GKLM) are also allowed. Key Manager keys migration is also supported (See KMIP integration guide for more details).

Question: What happens when external Key Manager connectivity goes down? Is my data accessible then?
Answer: Yes, data is still accessible when we cannot connect to the key manager since we store copy of the keys in the DD system also. New keys cannot be created, or key states cannot be synced when there is no connectivity with the external key manager.

Question: Is there a way that we can avoid storing keys in DD and store only in External Key Manager?
Answer: We always store a copy of the keys in the DD system for DIA purposes. This setting cannot be changed.

Question: Is there any performance impact due to the integration with KMIP?
Answer: No, there is no performance impact because of using external Key Managers.

Question: Is it possible to leverage the KMIP solution for selected Data Domains within the environment?
Answer: Yes, customers have complete flexibility in selecting the appropriate encryption methodology for their Data Domain systems. They can continue leveraging Data Domain’s embedded key manager on some Data Domain systems, and the encryption key rotation using KMIP on other Data Domain systems within their environment.

Question: Is the Data Domain to KMIP communication secure?
Answer: Yes, the Data Domain communicates over X509 certificate mutually authenticated sessions with the TLS. The user can use Data Domain CLI to import the appropriate X509 certificate into the Data Domain system. This certificate is then used to establish the secure channel between the DD and KMIP.

Key Life-Cycle Management 

Question: What key management capabilities exist with the DD Encryption option?
Answer: A key manager controls the generation, distribution, and life-cycle management of multiple encryption keys. A protection system can use either the embedded key manager or KMIP-complaint external key manager. Only one key manager can be in effect at a time. When encryption is enabled on a protection system, the Embedded Key Manager is in effect by default. If an External Key Manager is configured, it replaces the embedded key manager and remains in effect until it is disabled manually. Switching from Embedded Key Manager to External Key Manager or conversely, results in a new key being added to the system and it does not require a file system restart from release 7.1.

Question: What are the different key states on the Data Domain system?
The different key states on the Data Domain system are as follows:

• Activated-RW: There is only one key in this state on any DD system and is used for reading and writing data. This key is also used by the GC process to re-encrypt containers.
• Pending-Activated: There is only one key in this state on any DD system. This identified the key that will become Activated-RW after the next file system restart. Today, this state exists only at the time of enabling encryption. No other time pending-activated key is created.  
• Activated-RO: External key managers can have multiple Activated keys. The most recent key is in Activated-RW the rest are in this state. Keys may go into this state on the DD system when we cannot sync state with the key manager.
• Deactivated: This is used for reading existing data on the DD system.
• Compromised: When a customer compromises an external key manager key, then state will be changed to it after next key sync.  
• Marked-For-Destroyed: When a customer marks a key for destroy, then the key state becomes Marked-For-Destroyed. When GC is run, all containers encrypted with Marked-For-Destroyed keys are re-encrypted using the Activated-RW key.
• Destroyed: A key in the Marked-For-Destroyed state goes into this state when there is no data associated with it.
• Destroyed-compromised: A key in the Compromised state goes into this state when there is no data associated with it.

Question: Can encryption keys be exported for disaster recovery?
Answer: Keys can be exported manually using the below command.

"filesys encryption keys export"

DD system also exports keys by default when a new key gets added or when any key is deleted from the system.

Exported files are present in /ddr/var/.security and that is in an encrypted format. This file should then be copied off of the DDR and stored in a safe location which can be used in any disaster recovery condition later.

Note: Importing keys for disaster recovery requires Customer Support intervention as the restore process depends on the type of disaster encountered. We can import the exported key file using the following command.

filesys encryption keys import <filename> 

Question: Is the key generated by KMIP stored on the Data Domain system?
Answer: Yes, the encryption key obtained from the KMIP is stored in an encrypted fashion on the Data Domain system.

Question: How is a key state change in the KMIP appliance applied to the Data Domain system?
Answer: Key sync happens daily. If there is a new key available or state of key is changed, the sync updates the local key table. DD receives key updates from the KMIP everyday at midnight.

Question: Is it possible to manually synchronize the key states between the DD and KMIP?
Answer: Yes, the Data Domain CLI or UI can be used to manually synchronize the key states between the DD and KMIP. "filesys encryption keys sync" is the command used for it.

Question: Is it possible to change the time at which the DD receives key updates from KMIP?
Answer: No, it is not possible to change the time at which the DD receives key updates from the KMIP.

Question: Is there a limit to the number of keys supported by the Data Domain system?
Answer: Starting from DDOS 7.8, at any time, the Data Domain system can have a maximum of 1024 keys on the system. There is only one key in the Activated-RW and the rest of the keys could be in any other state.

Question: Can different keys be used for different datasets on Data Domain systems? Is this configurable?
Answer: No, we only support one active key in the system at a time and all incoming data are encrypted using the current active key. Keys cannot be controlled at a finer granularity like M-trees.

Question: Is there any notification when the maximum key limit is reached?
Answer: Yes, an alert is raised when the maximum key limit of 1024 is reached.

Question: How can I clear the alert about maximum key limit?
Answer: One of the keys must be deleted to clear the maximum key limit alert.

Question: Can we see the amount of data that is associated with a particular key on the Data Domain system?
Answer: Yes, this can be seen on DD but cannot be seen on the KMIP server. The user can use the Data Domain CLI and the UI to see the amount of data associated with a particular key.

Question:  Can we see age of keys on the DD system?
Answer: Yes, it can be seen on with EKM keys using UI.

Question: Does the old key work even if the time period has lapsed in making the new key effective?
Answer: There is no expiry date for encryption keys. Old keys become read-only keys after key rotation and stays in the DDOS.

Question: Does the encryption key automatically get deleted when there is no data associated with it on the Data Domain system?
Answer: No, the key is not automatically deleted. The user has to explicitly delete the key using the DD CLI or UI.

Question: Can a key be deleted even if there is data associated with it on the Data Domain system?
Answer: No, if a key has any data associated with it, then it cannot be deleted. Re-encryption of data is needed with some other key to delete a key that has data associated with it.

Question: If a key is deleted on the KMIP, then is the key also deleted from the Data Domain’s key list?
Answer: No, the user has to independently delete the key by using the DD CLI or UI.

Question: In a multi-site Data Domain environment, is a KMIP required at every location?
Answer: No, it is not necessary to have a KMIP at every site with a Data Domain. We can point to one KMIP server. It is recommended to have a separate key class for each DD system when they are using same KMIP server.

Question: If a key is compromised, do we have a process to retrieve the data that is encrypted with the old key?
Answer: In this case, the customer has to mark key as compromised in the KMIP server. Then do "filesys encryption keys sync" in DDOS system and then run "filesys encryption apply-changes" and then run GC (filesys clean). GC run re-encrypts all the data which was encrypted with the compromised key using a newer key. Once GC is done, the key state is changed to compromised-destroyed. Later they can delete that key. 

Encryption and Replication

Question: Is DD Replication supported and interoperable with the DD Encryption software option?
Answer: Yes, DD Replication can be used with DD Encryption option, thereby enabling encrypted data to be replicated using all different kinds of replication. Each replication form works uniquely with encryption and offers the same level of security.

Question: Do both the source and destination systems have to be running the same version of the DD OS to use encryption?
Answer: Source and destination can be in different DDOS version to use DARE with replication if replication compatibility matrix (see the admin guide for compatibility matrix) is in place. 

Question: How does replication work with encryption?
Answer: It depends on which form of replication is being used.

If the configured replication is MREPL or MFR:
If MREPL or MFR is used, DD Encryption can be licensed or enabled on either the source or destination independently depending on what the customer wants to achieve.

• When both source and destination have encryption enabled: When data ingested to source system, it is encrypted with the source system encryption key. The source decrypts the local data, re-encrypts it using the destination system encryption key, and then replicates the encrypted data to the destination when replicating.
• When the source has encryption disabled, and the destination has encryption enabled: All data ingested to source is not encrypted (for obvious reason.) But when replicating, the source encrypts the data using the destination system's encryption key and then replicates the encrypted data to the destination system. 
• When the source has encryption enabled, and the destination has encryption disabled: All data ingested to source system is encrypted with the source system's encryption key. The source decrypts the data and then replicates the unencrypted data to the destination Data Domain system when replicating.
• If encryption is enabled on the replica after replication context is set up, then any new segments now being replicated are encrypted at the source for the replica. Any segments residing on the replica prior to encryption being enabled are left in an unencrypted state unless encryption applies changes and GC run on the destination. 

If configured replication is CREPL:
With collection replication, both source and destination systems must be running same DDOS release. Therefore encryption must be either enabled or disabled on both. There cannot be a mismatch in encryption configuration also. Encryption keys are the same with source and destination. 

• When both source and destination have encryption enabled: All data ingested to the source system is encrypted using source system encryption key. When replicating, the source sends encrypted data to the destination Data Domain system in its encrypted state. The destination Data Domain system has the same key as the source since collection replication is all about an exact replica of the source Data Domain system. No data can be written to the destination Data Domain system outside of replication, as the destination is a read-only system.
• When both source and destination both have encryption disabled: All data ingested to source system is not encrypted (for obvious reason.) When replicating, the source sends (replicates) the data in an unencrypted state and it remains unencrypted at the destination Data Domain system. No data can be written to the destination Data Domain system outside of replication, as the destination is a read-only system.

Question: Is the destination's key stored indefinitely on the source Data Domain system?
Answer: The destination's encryption key is never stored on the source Data Domain system. It is only held in memory (encrypted) while the replication session is active. This applies to all kinds of replication except collection replication. In collection replication (CREPL) the same set of encryption keys are present in source and destination.  

Question: Can encryption be enabled in the CREPL system after replication context is established?
Answer: Yes, in this case encryption must be enabled in both source and destination. Encryption can be enabled in source and destination by disabling the replication context. Any new segments replicated are encrypted on the replica. Any segments residing on the replica prior to encryption being enabled are left in an unencrypted state.

Question: Can the DD Encryption be enabled concurrently with the over-the-wire encryption feature in the DD Replication software option?
Answer: Yes, both over the wire encryption and D@RE can be enabled concurrently to achieve different security goals.

Question: What happens if both the DD Encryption software option and the over-the-wire encryption feature in the DD Replication software option are enabled concurrently?
Answer: The first source encrypts data using destination encryption key. Then already encrypted data is encrypted a second time because of over the wire encryption while sending this data to its destination. At the destination after over the wire decryption is done, data will be stored in an encrypted format which was encrypted using destination's encryption key.

Question: When encryption is enabled in source and destination, do the passphrase must be the same on both?
Answer: If the configured replication is collection replication (CREPL), then passphrase must be the same. For other kinds of replication (like MREPL, MFR) passphrase can be different in source and destination.

Question: With encryption enabled on the destination (question not applicable to CREPL), do both the replicated data and data from some other access point (example through local backup) get encrypted? Is there a way to separate the two on the replica where only the replicated directories are encrypted while other access is not encrypted?
Answer: No, all data is encrypted at the replica(destination) regardless of entry point. Encryption cannot only be enabled or disabled at a MTree or directory level granularity. 

Question: How is the key exchange done between source and destination during MREPL or MFR?
Answer: During the association phase of replication, the target machine securely transmits its current encryption algorithm and key information to the source. Replication contexts are always authenticated with a shared secret. That shared secret is used to establish a "session" key using a Diffie-Hellman key exchange protocol. That session key is used to encrypt and decrypt the Data Domain system encryption key.

Question: What type of encryption algorithm is used for Data Domain's "encryption over the wire" feature, regarding encrypting the replication traffic?
Answer: When replication authentication-mode is set to "one-way" or "two-way," DHE (Ephemeral Diffie-Hellman) is used for session key exchange. Server authentication happens using RSA. AES 256-bit GCM cipher is used to encapsulate the replicated data over the wire. The encryption encapsulation layer is immediately removed when it lands on the destination system. 

One way indicates that only the destination certificate is certified. Two ways indicate that both the source and destination certificates are verified. Mutual trust must be established before you can use authentication-mode and both sides of the connection must enable this feature for encryption to proceed.

When replication authentication-mode is set to "anonymous," Anonymous Diffie-Hellman (ADH) is used for session key exchange, but in this case, source and destination do not authenticate each other before the key exchange. Also, if the authentication-mode is not specified, anonymous is used as the default.

Question: Does key rotation without file system restart work with all kinds of replication?
Answer: Key rotation without file system restart works with all sorts of replication except DREPL (DREPL support has already ended) and delta replication (which is also known as LBO)

Question: In the absence of certificates or PKI key pairs, how is the destination's encryption key protected during the key exchange?
Answer: There is a shared secret between all Data Domain replication pairs that is used to establish a shared session key using a Diffie-Hellman key exchange. That shared key is used to encrypt the destination's encryption key.

There is a difference between the shared secret used for replication authentication and the shared session key, which is allocated using the Diffie-Hellman key exchange protocol. The shared secret used for replication authentication is established by the Data Domain software the first time two Data Domain systems want to establish a replication context. It is also agreed upon through a Diffie-Hellman exchanged using parameters embedded in the code. This is persistently stored in the systems to authenticate every replication session between the two systems. The replication session key (the key used to encrypt the destination's encryption key) is established using another Diffie-Hellman exchange with the previously established shared secret, thereby driving the secure key exchange protocol. This key is not persistent and is only around while the replication context is active.

Question: Is it necessary for both Data Domains of a replication pair to use same external key manager (like KMIP key manager) solution or can one of the systems use external key manager and other can use embedded key manager?
Answer: Other than a collection replication pair, it is not necessary for both systems within a replication pair to use same key manager.

With collection replication, both Data Domain systems must be configured with the same key manager. But in that case also, the only source is syncing keys with the key manager and those keys are sent over to the destination also. With other replication types, different key managers can be used with source and destination.

Encryption and Migration

Question: Is data migration supported on systems with Encryption enabled?
Answer: Yes, data migration is supported on systems with encryption enabled. Encryption configuration on source and destination systems must be matched as a prerequisite before data migration is initiated. It is also recommended to export and backup encryption keys on the source system for DIA purposes before migration is initiated.

Question: Is data migration supported for both active tier and cloud tier migration for encryption enabled systems?
Answer: Yes, data migration is supported for both active tier and cloud tier migration for encryption-enabled systems. The list of pre-requisite attributes checked is applied based on which tier has encryption enabled.

Question: Which encryption settings are preserved as part of migration?
Answer: Encrypted data and encryption keys are migrated as is but settings like encryption key-manager, system passphrase, and other encryption configuration must be manually verified and matched for successful data migration. Any existing key-manager certificates are also transferred to the destination system. Encryption key-manager configuration has to be set up again on the destination system post migration.

Question: What encryption compatibility checks are done between source and destination during migration?
Answer: System passphrase, encryption status and key-manager configuration details, system FIPS mode settings are some of the Encryption settings that must be identical on source and destination systems for migration to be successful. This KB article 183040, Data Domain: Migration Procedure for Cloud Enabled DD Systems (log in to Dell Support required to view article), details the steps for migration between systems with cloud enabled. The same settings apply for active tier only migration as well.

Question: Is migration supported between systems with the Encryption Disablement Project setting on?
Answer: Data migration is supported between two systems if either both are EDP or both are non-EDP. Data migration is allowed from an EDP system to a non-EDP system if on-the-wire encryption is explicitly turned off. (using the MIGRATION_ENCRYPTION sysparam)

Encryption in Cloud Tier 

Question: Is Encryption supported for cloud tier?
Answer: Yes, encryption is supported on cloud tier. By default, encryption is disabled in the cloud tier. "cloud enable" command prompts to choose whether we want to enable encryption on the cloud tier.

Question: Are KMIP and External Key Managers supported with cloud tier?
Answer: Yes, KMIP and External Key Managers are supported with cloud tier from DDOS 7.8 onwards.

Question: At what granularity can encryption be enabled in cloud?
Answer: Encryption can be enabled and disabled on each cloud unit and each tier independently.

Question: Do cloud units have independent keys?
Answer: No, key management is common to both active and cloud tiers in DD. Keys are copied over to the respective unit/tier/cp when encryption is enabled. If encryption is enabled on active and not on cloud, active tier keys do not reflect on cloud and conversely. This applies to the cloud units too. (for example: cp1 has encryption enabled, and cp2 does not have encryption enabled then cp1 keys do not reflect on cp2.)   

Question: Can keys be deleted in the cloud?
Answer: No, deleting keys from the cloud is not supported.

Question: Where are data encryption keys managed for cloud units?
Answer: Keys are associated with a cp and each cloud unit is a different cp. A copy of keys from all cps is stored in active cp. 

Question: How do we recover cloud keys during disaster recovery? 
Answer: The cpnameval is mirrored to the cloud as part of the CP recovery, the encryption keys are going to be recovered to cpnameval. Now, we must run ddr_key_util tool to recover the keys.

Note: Disaster recovery requires Customer Support intervention.

Question: Can we do data movement when encryption is enabled only in cloud tier?
Answer: No, we must enable encryption in both cloud and active tiers for data movement.

Question: Can we enable External key-manager on cloud tier?
Answer: Yes, external key manager can be enabled on cloud tier. This feature is supported 7.8 onward. All the operations except destroy or delete key that is valid for active tier are also valid for cloud tier in terms of external key manager. 

Encryption and Garbage Collection

Question: What role does the Global Cleaning process play in Encryption-At-Rest; and is there a performance impact when enabling encryption for the first time?
Answer: Enabling encryption of data at rest for the first time has an impact on the performance of global cleaning. This is because data which must be read from existing containers on disk and written to new containers may require being read, decrypted, and uncompressed before being re-compressed, encrypted, and written back out to disk. When encryption is enabled on a DD holding a significant amount of preexisting data, and the 'filesys encryption apply-changes" command is run, then the subsequent global cleaning cycle attempts to encrypt all existing data on the system. This means that all data must be read, uncompressed, compressed, encrypted, and written out to disk. As a result the first global cleaning cycle after running 'filesys encryption apply-changes' may take longer than normal. Customers should ensure that they have sufficient free space on their DD system to allow cleaning to run to completion without the DD system becoming full (otherwise backups fail).

Question: Is there a performance impact to ongoing ingest/restore clean cycles?
Answer: Yes, there is performance impact and the impact generally depends on the amount of data being ingested/restored between clean cycles.

Question: How long does it take to encrypt my existing data?
Use this KB article to estimate the time, Data Domain: Calculating how long encryption at rest takes to apply.

Encryption and headswap 

Question: Can the customer move the disk to another Data Domain system if a head fails and access the disk when encryption is enabled?
Answer: The encryption key is not tied to the Data Domain system head itself, so you can move the disks to another Data Domain system head and the key is accessible there. On a new Data Domain system head, the file system is locked, so you have to enter the passphrase with the "filesys encryption unlock" command. 

Question: What if a customer forgot the passphrase at the time of headswap operation?
Answer: During that time, they can connect old head and work with support to reset the passphrase and then connect back to the new head and finish the headswap procedure. 

Encryption performance

Question: What is the observed impact on storage consumption when encryption is used?
Answer: It is negligible with roughly 1 percent overhead associated with storing some encryption parameters with user data.

Question: What is the observed impact on throughput (writes and reads) when Data at rest encryption is used?
Answer: The impact on ingest throughput when encryption is used can vary with the protocol and platform. In general, the following percentages are conservative performance degradations in aggregate throughput:

CBC Mode
  First Full: ~10% performance degradation on WRITEs
  Incremental: ~5% performance degradation on WRITEs
  Restores: 5 - 20% performance degradation on READs

GCM Mode
  First Full: 10 - 20% performance degradation on WRITEs
  Incremental: 5 -10% performance degradation on WRITEs
  Restores: 5 - 20% performance degradation on READs

These numbers are specific to the overhead of data at rest encryption (over the wire encryption is accounted separately)
 

Best practices

Question: What are best practices concerning key rotation policy?
Answer: Automated key rotation policy is not enabled by default. Customer has enabled it. It is recommended to rotate encryption keys frequently. When a system is configured with an external KMIP key manager, it is advisable to rotate keys frequently to handle any key compromise scenarios in future. When KMIP is configured with cloud tier, then suggested key rotation interval is 1 week and when KMIP is configured in only active tier then suggested key rotation policy is 1 month. However based on ingestion rate the, customer may increase or decrease key rotation frequency too. If embedded key manager is configured, then key rotation policy anywhere between 1 - 3 months is recommended. 

Question: What are best practices with KMIP key class if a customer using same KMIP server for many DD systems?
Answer: It is recommended to have a separate key class for each DD system when they are using same KMIP server. By that way key rotation done in one DDOS system does not impact the state of the key present in other DDOS system.

For further information, see the document DELL EMC PowerProtect DD Series Appliances: Encryption Software (delltechnologies.com)

Encryption: Technical White Paper PowerProtect DD Series Appliances: Encryption Software

Other documentation related to DD Encryption (Admin Guide, Command Reference Guide, and Security Config Guide) can be found in article 126375, PowerProtect and Data Domain core documents.

KMIP Integration Guide and Compatibility Matrix

Watch this video: