Data Protection Advisor: How to upgrade to DPA 19.5 or later in a Linux environment when Datastore Replication is configured

References to non-inclusive language have been modified and replaced with "Primary" and "Secondary".

Upgrading to DPA 19.5 or later in a Linux environment when Datastore Replication is configured.

These steps are a workaround for the known issues with performing a standard DPA 19.5 or later upgrade, as documented in the KB article 191139: Data Protection Advisor: Datastore Replication is nonfunctional in upgrade or new configuration of version 19.5. 

These steps apply to the upgrade of an environment with DPA DataStore (DS) replication configured (in use) from DPA 19.4 or earlier to DPA 19.5 or later.

Preface:

There are two database users of interest while performing these upgrade steps. These two Users are:

• apollouser
  • This user is called the "database user" and is set during installation of DPA. The installer prompts for the password for the database user. This is that user.
  • This user's password can be reset from DPA  using the CLI command below:

• apollosuperuser
  • This user is called the "database superuser." This user owns the database and has all privileges. This user is not set during installation of DPA. The installer does not prompt for the password for this user. The password is instead set to a strong, randomized password that is different on every installation of the DPA Datastore.
  • This user's password can be reset from DPA using CLI command below:

Both of these database users and these commands are outlined in further detail in the DPA Installation and Administration Guide.

With Datastore Replication, the database user's passwords are important. For Datastore Replication setups, the passwords for apollouser and apollosuperuser must be identical on both the Primary Datastore and Secondary Datastore. They must be set and known by the DPA environment administrator.

It was observed that not all DPA administrators routinely set the apollosuperuser password on the DPA Datastore, whether it is a Standalone Datastore, Primary Datastore, or Secondary Datastore. This leads to there being two scenarios and sets of steps for upgrading a DPA environment with Datastore Replication.

Instructions:

Scenario 1: The DPA apollosuperuser password is not known or not set on the Primary and Secondary Datastores

1. Stop all DPA services on all DPA nodes:

• Run command:  dpa.sh app stop  on the Application Server.
• Run command:  dpa.sh ds stop  on the Primary Datastore.
• Run command:  dpa.sh ds stop on the Secondary Datastore.

2. Upgrade Primary Datastore:

• Launch the DPA installer and upgrade the Primary Datastore.
• Verify that the Primary Datastore is running.
  • Run command: dpa.sh ds status

3. Stop the Primary Datastore:

• Run command: dpa.sh ds stop

4. On the Primary Datastore, modify the Postgres configuration file:

• Go to the <dpa_install_path>/datastore/data directory
• Make a backup copy of the file named postgresql.conf in case it is required to revert to.
• Edit the postgresql.conf file.
• Search or scroll and look for the line that says:
  • max_wal_senders = # 
• Where the # above is a numeric value (example, 1, 2, and so forth).
• Modify the numeric value to 10 so it appears as below: 
  • max_wal_senders = 10
• Save the postgresql.conf file.

5. Start the Primary Datastore service:

• Run command: dpa ds start

6. Create a replication slot for Datastore Replication:

• It is necessary to manually create the Datastore Replication slot on the Primary Datastore only. This is required for the Datastore Replication configuration to function.
• From the DPA CLI on the Primary Datastore, run the following database query.
  • dpa.sh ds query "select * from pg_create_physical_replication_slot('standby1_slot');"
• Below is an example of the command that is run and the expected output:

7. Verify that the replication slot was created:

• From the DPA CLI on the Primary Datastore, run the following database query.
  • dpa.sh ds query "select slot_name, slot_type, active, wal_status from pg_replication_slots;"
• Below is an example of the command that is run and the expected output:

 

8. Set the apollosuperuser password on the Primary Datastore:

• From the DPA CLI on the Primary Datastore, run the following command:
  • dpa.sh ds superpwd
• Follow the prompts to set the password with the requirements listed.
• Keep note of the password that was set.
• Below is an example of the command that is run and expected output:

9. Upgrade the DPA Application

• Launch the DPA installer and upgrade the DPA Application.
• Verify that the DPA Application is running:
  • Run command: dpa app status
• Initially the DPA Application status reports as starting for a period of time and this is normal. The length of the period of time varies per environment and size of the database, but typically can be from 15 minutes to 1-2 hours. During this period of time, the DPA Application is running upgrade scripts to upgrade the contents of the Primary Datastore (database). Do not interrupt this process and allow it to complete. Once complete, the status of the DPA Application reports as running.

10. Launch the DPA UI and verify connectivity and functionality:

• Log in to the DPA UI as a valid user (or a user with administrator privileges).
• Verify that the functionality in the DPA UI is as expected.
• If required, verify that Agent connections and data collection processing are working as expected. Data collection may take an undetermined period of time to catch up as the DPA Application and Primary Datastore have been offline during the upgrade process.

Checkpoint: If DPA is functioning properly, go to the next steps.
 

11. Stop the DPA Application:

• Run command: dpa app stop

Steps 12 to 13 and steps 14 to 17 can be run concurrently (simultaneously).
 

12. Create a Datastore Replication Export on the Primary Datastore:

• From the DPA CLI on the Primary Datastore, run the following command:

  • dpa.sh ds rep -e <replication_export_directory>

• The replication_export_directory must be created beforehand. Also this directory requires sufficient space to store the replication export.

• The replication export takes a period of time to complete. This may be in the order of hours depending on several factors including overall database size.

• It is advisable to keep the DPA Application shut down during the replication export and also during the transfer of the replication export to the Secondary Datastore and ideally during the import on the Secondary Datastore.

[root@ds01]# /opt/emc/dpa/services/bin/dpa.sh ds rep -e rep_export

EMC Data Protection Advisor

Exporting Datastore....

Datastore base replica created in directory rep_export

Command completed successfully.

13. Copy the replication export from the Primary Datastore to the Secondary Datastore.
14. Uninstall the Secondary Datastore.
15. Install DPA on the Secondary Datastore using the same upgrade version that was used for the DPA Application and Primary Datastore in the earlier steps.

• The version and build for installation should match the version that is used for upgrading the DPA Application and Primary Datastore.

• The installation should be as a Standalone Datastore server and point to the DPA Application.

• During installation, when prompted for passwords, ensure all passwords that are entered are identical to the passwords used on the Primary Datastore.

16. Stop the Standalone (original Secondary) Datastore service:

• Run command: dpa.sh ds stop

17. Change the role of the Standalone (original Secondary) Datastore to a Secondary Datastore.

• From the DPA CLI on the Datastore, run the following command:

  • dpa.sh ds rep --role SLAVE <IP_Address_of_Primary_Datastore>

• Below is an example of the command that is run and expected output:

[root@ds02]# /opt/emc/dpa/services/bin/dpa.sh ds rep --role SLAVE 192.168.0.1

EMC Data Protection Advisor

Command completed successfully.

18. Import the Datastore Replication export from the Primary Datastore into the Secondary Datastore.

• From the DPA CLI on the Secondary Datastore, run the following command

  • dpa.sh ds rep -i <replication_export_directory>

• The replication_export_directory is where the Datastore Replication export from the Primary Datastore was copied to on the Secondary Datastore.

• The replication import takes a period of time to complete. This may take minutes or hours depending on several factors including overall database size. It has been observed that often the import can take less time than the export took.

[root@ds02]# /opt/emc/dpa/services/bin/dpa.sh ds rep -i ./rep_export

EMC Data Protection Advisor

[INFO] Configuration files copied away.

[INFO] Data directory deleted.

[INFO] Datastore baseline replica restored into datastore

[INFO] Configuring Datastore. Please wait, this may take few minutes.

Command completed successfully.

19. Create a standby.signal file on the Secondary Datastore.

• Go to the <dpa_install_path>/datastore/data directory.

• Search for a file called  standby.signal

• If this file does not exist, create the file that is named standby.signal This can be done with the touch command, as below.

  • touch standby.signal

20. Start the Secondary Datastore service.

• Run command: dpa.sh ds start

21. Set the apollosuperuser password on the Secondary Datastore.

• From the DPA CLI on the Secondary Datastore, run the following command:

  • dpa.sh ds superpwd

• Follow the prompts to set the password with the requirements listed.

• IMPORTANT: This password must be the same apollosuperuser password that was set on the Primary Datastore.

• Below is an example of the command that is run and expected output:

[root@ds02 ~]# /opt/emc/dpa/services/bin/dpa.sh ds superpwd

EMC Data Protection Advisor

Enter new password for the superuser owning the database.

The password must have:

    - at least 9 characters

    - at least 1 uppercase letter

    - at least 1 lowercase letter

    - at least 1 special character

    - at least 1 digit

Retype new password for the superuser owning the database :

[INFO] New password has been applied to the superuser owning the database.

Command completed successfully.

22. Verify that Datastore Replication is working correctly.

• From the DPA CLI on the Primary Datastore, run the following command:

  • dpa.sh ds rep status

• The status of the replication should be streaming

• Below is an example of the command that is run and expected output:

[root@ds01]# /opt/emc/dpa/services/bin/dpa.sh ds rep stat

EMC Data Protection Advisor

[INFO] Replication State : MASTER

[INFO] Defined Slaves

                         : 192.168.0.2/32

[INFO]                                    SLAVE       BYTES LAG      STATUS

[INFO]                           192.168.0.2               0      streaming

Command completed successfully.

• Sometimes the status of the replication may be catch-up . This is normal as the databases may need time to completely sync (catch-up) changes. Allow this to continue and check periodically that the status eventually moves to streaming.

23. Start the DPA Application:

• Run command: dpa app start

Scenario 2: The DPA apollosuperuser password is known and set identically on the Primary and Secondary Datastores.

1. Stop all DPA services on all DPA nodes:

• Run command:  dpa.sh app stop  on the Application Server.
• Run command:  dpa.sh ds stop  on the Primary Datastore.
• Run command:  dpa.sh ds stop on the Secondary Datastore.

2. Upgrade Primary Datastore:

• Launch the DPA installer and upgrade the Primary Datastore.
• Verify that the Primary Datastore is running.
  • Run command: dpa.sh ds status

3. Stop the Primary Datastore:

• Run command: dpa.sh ds stop

4. On the Primary Datastore, modify the Postgres configuration file:

• Go to the <dpa_install_path>/datastore/data directory
• Make a backup copy of the file named postgresql.conf in case it is required to revert to.
• Edit the postgresql.conf file.
• Search or scroll and look for the line that says:
  • max_wal_senders = # 
• Where the # above is a numeric value (example, 1, 2, and so forth).
• Modify the numeric value to 10 so it appears as below:
  • max_wal_senders = 10
• Save the postgresql.conf file.

5. Start the Primary Datastore service:

• Run command: dpa ds start

6. Create a replication slot for Datastore Replication:

• It is necessary to manually create the Datastore Replication slot on the Primary Datastore only. This is required for the Datastore Replication configuration to function.
• From the DPA CLI on the Primary Datastore, run the following database query.
  • dpa.sh ds query "select * from pg_create_physical_replication_slot('standby1_slot');"
• Below is an example of the command that is run and expected output:

[root@ds01 ~]# /opt/emc/dpa/services/bin/dpa.sh ds query "select * from pg_create_physical_replication_slot('standby1_slot');"

EMC Data Protection Advisor

   slot_name   | lsn

---------------+-----

 standby1_slot |

(1 row)

7. Verify that the replication slot was created:

• From the DPA CLI on the Primary Datastore, run the following database query.
  • dpa.sh ds query "select slot_name, slot_type, active, wal_status from pg_replication_slots;"
• Below is an example of the command that is run and expected output:

[root@ds01 ~]# /opt/emc/dpa/services/bin/dpa.sh ds query "select slot_name, slot_type, active from pg_replication_slots;"

EMC Data Protection Advisor

   slot_name   | slot_type | active 

---------------+-----------+--------

 standby1_slot | physical  | f      

(1 row)

8. Upgrade the DPA Application

• Launch the DPA installer and upgrade the DPA Application.
• Verify that the DPA Application is running:
  • Run command: dpa app status
• Initially the DPA Application status reports as starting for a period of time and this is normal. The length of the period of time varies per environment and size of the database, but typically can be from 15 minutes to 1-2 hours. During this period of time, the DPA Application is running upgrade scripts to upgrade the contents of the Primary Datastore (database). Do not interrupt this process and allow it to complete. Once complete, the status of the DPA Application reports as running.

9. Launch the DPA UI and verify connectivity and functionality:

• Log in to the DPA UI as a valid user (or a user with administrator privileges).
• Verify that the functionality in the DPA UI is as expected.
• If required, verify that Agent connections and data collection processing are working as expected. Data collection may take an undetermined period of time to catch up as the DPA Application and Primary Datastore have been offline during the upgrade process.

Checkpoint: If DPA is functioning properly, go to the next steps.
 

10. Stop the DPA Application:

• Run command: dpa app stop

Steps 11 to 12 and steps 13 to 16 can be run concurrently (simultaneously).
 

11. Create a Datastore Replication Export on the Primary Datastore:

• From the DPA CLI on the Primary Datastore, run the following command:
  • dpa.sh ds rep -e <replication_export_directory>
• The replication_export_directory must be created beforehand. Also this directory requires sufficient space to store the replication export.
• The replication export takes a period of time to complete. This may be in the order of hours depending on several factors including overall database size.
• It is advisable to keep the DPA Application shut down during the replication export and also during the transfer of the replication export to the Secondary Datastore and ideally during the import on the Secondary Datastore.

[root@ds01]# /opt/emc/dpa/services/bin/dpa.sh ds rep -e rep_export

EMC Data Protection Advisor

Exporting Datastore....

Datastore base replica created in directory rep_export

Command completed successfully.

12. Copy the replication export from the Primary Datastore to the Secondary Datastore.
13. Change the role of the Secondary Datastore to a Primary Datastore.

• From the DPA CLI on the Datastore, run the following command:
  • dpa.sh ds rep --role MASTER
• Below is an example of the command that is run and expected output:

[root@ds02]# /opt/emc/dpa/services/bin/dpa.sh ds rep --role MASTER 

EMC Data Protection Advisor

Command completed successfully.

14. Upgrade Secondary Datastore:

• Launch the DPA installer and upgrade the Secondary Datastore.
• Verify that the Secondary Datastore is running.
  • Run command:  dpa.sh ds status

15. Stop the Secondary Datastore service:

• Run command:  dpa.sh ds stop

16. Change the role of the Secondary Datastore back to a Secondary Datastore.

• From the DPA CLI on the Datastore, run the following command:
  • dpa.sh ds rep --role SLAVE <IP_Address_of_Primary_Datastore>
• Below is an example of the command that is run and expected output:

[root@ds02]# /opt/emc/dpa/services/bin/dpa.sh ds rep --role SLAVE 192.168.0.1

EMC Data Protection Advisor

Command completed successfully.

17. Import the Datastore Replication export from the Primary Datastore into the Secondary Datastore.

• From the DPA CLI on the Secondary Datastore, run the following command:
  • dpa.sh ds rep -i <replication_export_directory>
• The replication_export_directory is where the Datastore Replication export from the Primary Datastore was copied to on the Secondary Datastore.
• The replication import takes a period of time to complete. This may take minutes or hours depending on several factors including overall database size. It has been observed that often the import can take less time than the export took.

[root@ds02]# /opt/emc/dpa/services/bin/dpa.sh ds rep -i ./rep_export

EMC Data Protection Advisor

[INFO] Configuration files copied away.

[INFO] Data directory deleted.

[INFO] Datastore baseline replica restored into datastore

[INFO] Configuring Datastore. Please wait, this may take few minutes.

Command completed successfully.

18. Create a standby.signal file on the Secondary Datastore.

• Go to the <dpa_install_path>/datastore/data directory.
• Search for a file called  standby.signal
• If this file does not exist, create the file that is named standby.signal. This can be done with the touch command, as below:
  • touch standby.signal

19. Start the Secondary Datastore service.

• Run command: dpa.sh ds start

20. Verify that Datastore Replication is working correctly.

• From the DPA CLI on the Primary Datastore, run the following command:
  • dpa.sh ds rep status
• The status of the replication should be streaming
• Below is an example of the command that is run and expected output:

[root@ds01]# /opt/emc/dpa/services/bin/dpa.sh ds rep stat

EMC Data Protection Advisor

[INFO] Replication State : MASTER

[INFO] Defined Slaves

                         : 192.168.0.2/32

[INFO]                                    SLAVE       BYTES LAG      STATUS

[INFO]                           192.168.0.2               0      streaming

Command completed successfully.

• Sometimes the status of the replication may be catch-up . This is normal as the databases may need time to completely sync (catch-up) changes. Allow this to continue and check periodically that the status eventually moves to streaming.

21. Start the DPA Application:

• Run command: dpa app start

 

NOTE: These steps apply to a Datastore Replication environment with a single Application server, a Primary Datastore and a Secondary Datastore. Other variations with the Datastore Replication environment are possible. These environment variations will require some modifications and/or additions to the steps. For help with these variations contact Dell Technical Support.

Contact Dell Technical Support for further assistance, details, or information.