Cloud Disaster Recovery: Amazon Web Services (AWS) deprecation of Python 2.7 support as of July 15, 2021 may impact Cloud Disaster Recovery customers, which may result in potential data unavailability

Starting July 15, 2021, AWS has deprecated Python 2.7 support in AWS CLI, causing the following issues for Cloud DR:

• New deployment of all Cloud DR versions
• Existing Deployments:
  • Running recovery on all Cloud DR versionsRetention on versions 19.7 and 19.8 

What is Impacted?
Products:
All Cloud DR associated products are impacted by this issue (PowerProtect DP Appliances, PPDM, Standalone Cloud DR, and RecoverPoint for VM).

Customers:
All Cloud DR customers using AWS or AWS GovCloud as their target cloud are impacted.

Functionality:
New Cloud DR installations:

• Fail to deploy the CDRS in AWS or AWS GovCloud (impacting all Cloud DR releases).

Existing Cloud DR deployments: 

• Fail to run recovery flows in AWS or AWS GovCloud (impacting all Cloud DR releases).
• Fail to run retention flow in AWS or AWS GovCloud (impacting 19.7 and 19.8 Cloud DR releases).

What’s Not impacted:

• Existing Cloud DR deployments can still successfully run protection to AWS or AWS GovCloud.
• Cloud DR customers using Azure and Azure Government clouds are not impacted.

Workaround:
New Cloud DR Server deployment into AWS or AWS Govcloud requires using a new Installation package of one of the new patches released.

The issue is due to AWS discontinued Python 2.7 support. See Amazon EOS information:
https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-python-2-7-in-aws-sdk-for-python-and-aws-cli-v1/.

Here are the two resolution options for the impacted customers; Table #1 offers new deployment and upgrade patches, Table #2 offers workaround as an alternative to customers who do not choose to upgrade.

Table 1

Product (Operating System or Operating Environment)	Cloud Disaster Recovery (Cloud DR) version 19.8 and earlier with AWS or AWS GovCloud All Cloud DR associated products with AWS or AWS GovCloud may be impacted by this issue PowerProtect DP Appliances or Integrated Data Protection Appliance (IDPA) PowerProtect Data Manager (PPDM) Standalone Cloud DR (PowerProtect DD or Data Domain and Avamar) RecoverPoint for VMs
Issue Addressed in this OS, OE or Software	Cloud DR 19.8.0.2, 19.7.0.2, 19.6.0.2, 19.5.0.2, 19.2.0.5, and 19.1.0.6
ACTION TYPE	UPGRADE
Who Can Complete the Action	CUSTOMER
Resolution Detail	For Cloud DR New Deployment: Upgrade (or use) Cloud DR 19.8.0.2, 19.7.0.2, 19.6.0.2, 19.5.0.2, or 19.2.0.5 For Cloud DR PowerProtect or IDPA Customers: IDPA Release 2.6.1 customers: Upgrade to Cloud DR 19.6.0.2 IDPA Release 2.6.0 customers: Upgrade to Cloud DR 19.5.0.2 IDPA Release 2.5.0 customers: Upgrade to Cloud DR 19.2.0.5 IDPA Release 2.4.1 and 2.3.1 customers: Upgrade to Cloud DR 19.1.0.6 Note: IDPA Release 2.4.0 and 2.3.0 customers: See ‘WORKAROUND’ section under Table #2 (re-enablement of recovery flows only) For Cloud DR PPDM Customers: PPDM Release 19.8 customers: Upgrade to Cloud DR 19.8.0.2 PPDM Release 19.7 customers: Upgrade to Cloud DR 19.7.0.2 PPDM Release 19.6 customers: Upgrade to Cloud DR 19.6.0.2 PPDM Release 19.5 customers: Upgrade to Cloud DR 19.5.0.2 PPDM Release 19.2 customers: Upgrade to Cloud DR 19.2.0.5 PPDM 19.3 and 19.4 customers: Upgrade to Cloud DR 19.5.0.2 (Cloud DR Server instance) For Cloud DR RecoverPoint for VM Customers: RecoverPoint for VM Release 5.3.2 customers: Upgrade to Cloud DR 19.7.0.2 RecoverPoint for VM Release 5.3.1 customers: Upgrade to Cloud DR 19.6.0.2 RecoverPoint for VM Release 5.2.x customers: Upgrade to Cloud DR 19.5.0.2 For standalone Cloud DR customers: Available Upgrade Patches for Standalone Cloud DR: 19.8.0.2, 19.7.0.2, 19.6.0.2, 19.5.0.2, 19.2.0.5, and 19.1.0.6  Standalone Cloud DR 19.3 and 19.4 impacted customers should upgrade to Cloud DR 19.5.0.2 release or later All Cloud DR installation or upgrade packages are available at: https://www.dell.com/support/home/en-in/product-support/product/data-domain-cloud-dr-for-avamar/drivers

Table 2
Workaround for any customer who does not choose to apply the Cloud DR patches detailed in Table #1.
 

Product (Operating System or Operating Environment)	Cloud Disaster Recovery (Cloud DR) version 19.8 and earlier with AWS or AWS GovCloud All Cloud DR associated products with AWS or AWS GovCloud may be impacted by this issue PowerProtect DP Appliances PowerProtect Data Manager (PPDM) Standalone Cloud DR (PowerProtect DD or Data Domain and Avamar) RecoverPoint for VMs
ACTION TYPE	WORKAROUND
Who Can Complete the Action	CUSTOMER
Resolution Detail	WORKAROUND: Avoid removing Cloud DR Server (CDRS) or any Cloud DR components running in the cloud. New Cloud DR Server Deployment Workaround details: Workaround for PowerProtect DP / IDPA Appliances, PowerProtect Data Manager and RecoverPoint for VMs customers running existing versions that do not include the fixed Cloud DR patches to enable Cloud DR Server deployment to AWS and AWS GovCloud Workaround details: Using an external Cloud DR Add-On (CDRA) in order to successfully deploy CDRS to AWS or AWS GovCloud and later connect PowerProtect DP appliance, PPDM or RecoverPoint for VM to that CDRS instance. Steps for this workaround: Deploy CDRA VM using new patch Installation package of needed Cloud DR release. From CDRA UI, deploy CDRS in the required AWS or AWS GovCloud region. Connect PowerProtect DP appliance, PPDM, or RecoverPoint for VM with CDRS. Remove the temporary standalone CDRA VM: From the CDRS System menu option, select Registered Components. Select the temp CDRA and click Unregister. Power off the temp CDRA VM. Delete the temp CDRA VM. Workaround for customers that cannot upgrade their existing CDRS using the released patches to re-enable Cloud DR recovery workflows: Clean all existing messages from the relevant SQS queues: Purge Console messages: Log in to AWS Console. Select the appropriate Region. Go to Services > SQS (Simple Queue Service). Select the RestoreService queues (starting with CDRS-RestoreService) where Messages available is greater than zero (one by one). Click on action and purge them to clear all messages from queue. Write “purge” in the confirmation box and click the Purge button. Terminate all RestoreService instances: Log in to AWS Console. Select the appropriate Region. Go to Services > EC2 (Amazon EC2) > Auto Scaling groups. Filter the Auto Scaling groups (starting with CDRS-RestoreService). Select the Auto Scaling group which has Desired capacity is greater than zero. Click on Edit button, set the Desired capacity to zero, and click on Update button. This terminates all the RestoreService instances. Update the RestoreService CF stack to use a specific CLI v1 version: Log in to AWS Console. Select the appropriate Region. Go to Services > CloudFormation > Stacks. Unselect the toggle button View Nested. Filter out the impacted RestoreService stack (starting with CDRS-RestoreService). Click on the CDRS-RestoreService stack and go to the Parameters section. Copy the InstanceUserData encrypted string into an online decryption tool (for example, https://www.base64decode.org/) and decode this string. Change the following two lines in the decrypted script: FROM     wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip     unzip awscli-bundle.zip  TO     wget https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip     unzip awscli-bundle-1.18.200.zip Encode (using https://www.base64encode.org/) the updated complete decoded script back to the base64 and copy the encrypted string. Go to AWS Console and click the Update button in the stack and select Use Current Template radio option, click Next. In Parameters, replace the InstanceUserData field value with the new encrypted string (copied in step i). Click Next, click Next again, check the acknowledgments checkboxes, and then click Update Stack button. Save the change and wait for the stack to finish its internal update process. It should update all the nested stacks as part of this step. Verify that DR flows and Rapid Recovery copies creation can now progress and complete successfully. Retention should succeed.