VxRail: Switching from CloudLink to Native Key Provider

How to Switch from CloudLink to Native Key Provider:

Backups: As a precaution, take backups and snapshots of CloudLink and vCenter.

1. You can take file-based backups of vCenter from within the VAMI: https://vCenterIP:5480
2. Take backups of CloudLink in the CloudLink UI > SYSTEM> Backup > Generate New Backup and then Actions > Download Backup.
3. Ensure that you are able to locate the CloudLink backup files ccbackup.bak and the CloudLink backup key cckey.pem
4. The cckey would have been downloaded during initial deployment of CloudLink. If unable to locate the cckey, you can generate a new key by going to SYSTEM > Backup > Actions > Generate and Download New Key.

• Generating a new key makes previous backups invalid and only new backups are valid
• If generating a new cckey, take new backups immediately by going to SYSTEM > Backup >  Generate New Backup, and then Actions > Download Backup  

ccbackup-d5545c097e4e4h8244b487b4d6d7372c-2024-12-22_06-31-58.bak    <----- The backup file will look similar to this
cckey-d5545c097e4e4h8244b487b4d6d7372c.pem                           <----- The backup key will look similar to this

5. In the CloudLink UI > SYSTEM > Vault, check if the Vault Unlock Mode is set to auto or manual. If it is set to manual, ensure that you know the unlocked codes that were configured during deployment of CloudLink. In the CloudLink UI > ABOUT > Rest Documentation, there is a restapi command called Validate passcode inside the lockbox section that you can use to check the unlocked codes.
6. Switching from CloudLink to NKP does not require any downtime. The steps below perform a shallow rekey where new keys get created on the native key provider. This happens quickly as it does not perform a full reencryption of the data. The process is going to be different depending on whether running vSAN encryption or VM encryption. This can be done when hosts and VMs are powered on.

How to identify encryption KeyIDs:

• With vSAN encryption, every ESXi host in the cluster shares the same Hostkey and KEK. Run this command on encrypted ESXi hosts when doing vSAN encryption: 

esxcli vsan encryption info get

• With VM encryption, each host in the cluster has a different HostKey. Run this command on encrypted ESXi hosts when doing VM encryption: 

crypto-util keys getkidbyname HostKey

• Run this PowerCLI command to get the Encryption KeyID of all encrypted VMs and the Key Provider the VMs belongs to:

get-vm | Get-SecurityInfo | Where-Object { $_.isencrypted -eq $true} | Select-Object parent, keyproviderid, isencrypted, isencryptionlocked, encryptionkeyid | Format-Table -Wrap -Autosize

• Locate all the encryption keys in the CloudLink UI:

KMIP SERVER > Partitions > select a partions > Actions > Show Objects

• This PowerCLI script list the HostKey of each ESXi host and the KeyProvider it is using:

$vmhosts = Get-VMHost

$results = foreach ($vmhost in $vmhosts) {
    $runtime = $vmhost.ExtensionData.Runtime

    if ($runtime.CryptoKeyId) {
        $keyId = $runtime.CryptoKeyId.KeyId
        $providerId = $runtime.CryptoKeyId.ProviderId.Id
        $cryptoState = $runtime.CryptoState

        [PSCustomObject]@{
            HostName     = $vmhost.Name
            ProviderId   = $providerId
            CryptoState  = $cryptoState
            KeyId        = $keyId  
        }
    } else {
        [PSCustomObject]@{
            HostName     = $vmhost.Name
            ProviderId   = "None"
            CryptoState  = $runtime.CryptoState
            KeyId        = "None"
        }
    }
}
$results | Format-Table -Wrap -AutoSize

Switching from CloudLink to Native Key Provider on a cluster with vSAN data-at-rest encryption enabled:

1. Create a Native Key Provider:

vCenter > Configure > Key Providers > Add > Add Native Key Provider

• When NKP is first created, the option to select "Use key provider only with TPM protected ESXi hosts" shows
• Using TPM with Native Key Provider is not required, but it is recommended. If enabling this option, verify that TPM 2.0 is on every host in the cluster
• Check the TPM version in the iDRAC in:

Configure > BIOS Settings > System Security > TPM.

• Find the TPM version in vCenter by:

selecting the cluster > Monitor > Security.

• TPM 1.2 is deprecated. Confirm that each host has TPM version 2.0 if enabling this feature. If using TPM, ensure that the TPM recovery keys are saved. This lists the commands to display the recovery keys: How to Gather Recovery Key from TPM Security Enabled Hosts

2. Backup of the Native Key Provider Backing up native key provider is required before it can be used.
3. Set the Native Key Provider as the Default Key Provider in vCenter:

vCenter > Configure > Key Provider

4. Run the command below on the encrypted ESXi hosts prior to generating new encryption keys:

esxcli vsan encryption info get

5. In vCenter, select the encrypted cluster:

Select the encrypted cluster > Configure > vSAN/Services > Data Services > Edit > Change to the new Key Provider and hit Apply

6. Run the command below on the encrypted ESXi hosts to verify new keys were generated on the Native Key Provider.

esxcli vsan encryption info get

7. If there are any encrypted VMs on this same cluster, generate new VM keys switching from CloudLink to Native Key Provider. These are VMs that are in an encrypted storage policy or VMs that have a vTPM. This is done by Right-clicking all the encrypted VMs and Re-Encrypting: 

Right-clicking all the encrypted VMs > VM Policies > Re-Encrypt

8.  Run the PowerCLI command before and after reencrypting the VMs to confirm the keyID has changed. It should also show that the VM is associated with the correct Key Provider:

get-vm | Get-SecurityInfo | Where-Object { $_.isencrypted -eq $true} | Select-Object parent, keyproviderid, isencrypted, isencryptionlocked, encryptionkeyid | Format-Table -Wrap -Autosize

Switching from CloudLink to Native Key Provider on a cluster using VM encryption (VMs that are in an encrypted storage policy or VMs that have a vTPM) with vSAN data-at-rest encryption disabled:

1. Create a Native Key Provider in vCenter:

vCenter > Configure > Key Providers > Add > Add Native Key Provider

• When NKP is first created, the option to select 'Use key provider only with TPM protected ESXi hosts' is visible
• Using TPM with Native Key Provider is not required, but it is recommended. If enabling this option, verify TPM 2.0 is on every host in the cluster
• Check the TPM version in iDRAC:

Configure > BIOS Settings > System Security > TPM.

• The TPM version is also visible in vCenter:

vCenter: selecting the cluster > Monitor > Security.

• TPM 1.2 is deprecated. Confirm that each host has TPM version 2.0 if enabling this feature. If using TPM, ensure that the TPM recovery keys are saved. This article lists the commands to display the recovery keys: How to Gather Recovery Key from TPM Security Enabled Hosts

2. Take a backup of the Native Key Provider. Backing up the Native Key Provider is required before it can be used
3. Set the Native Key Provider as the Default Key Provider in vCenter:

vCenter > Configure > Key Providers

4. Locate all the encrypted VMs. These are VMs that are in an encrypted storage policy or VMs that have a vTPM. 
5. Run the PowerCLI command below to identify Encryption KeyID of the VMs:

get-vm | Get-SecurityInfo | Where-Object { $_.isencrypted -eq $true} | Select-Object parent, keyproviderid, isencrypted, isencryptionlocked, encryptionkeyid | Format-Table -Wrap -Autosize

6. Generate a new encryption key for all encrypted VMs:

Right-click all the encrypted VMs > VM Policies > Re-Encrypt

7. Run the PowerCLI command below to verify the Encryption KeyID of the VMs have changed and that the VM is associated with the correct Key Provider. 

get-vm | Get-SecurityInfo | Where-Object { $_.isencrypted -eq $true} | Select-Object parent, keyproviderid, isencrypted, isencryptionlocked, encryptionkeyid | Format-Table -Wrap -Autosize

8. Run the command below on the hosts before generating new encryption keys.

crypto-util keys getkidbyname HostKey

9. Each ESXi host has an encryption key that must be migrated and or created on the Native Key Provider. This can be done either using PowerCLI or from the https://vCenter/mob 

• Use PowerCLI to generate new ESXi encryption keys (these commands do not work if a host is in maintenance mode0. 

1) get-vmhost
2) get-KeyProvider
3) $keyprovider = Get-KeyProvider | Where-Object { $_.Name -eq "keyprovidername" }
4) $keyprovider
5) Set-VMHost -VMHost esxihostname -KeyProvider $keyprovider

• Use vCenter mob to generate new ESXi encryption keys. 
• Rekey the host key with HostSystem.ConfigureCryptoKey method

10. If successful, the task Configure the host key is completed in vCenter recent tasks.
11. Run this command on the hosts to verify a new key was generated on the Native Key Provider.

crypto-util keys getkidbyname HostKey