IDPA Upgrade to 2.3 fails on Search with error "check_search_version_on_master() Could not verify Search version"

Summary: This KB walks you through Search Upgrade issue seen on IDPA 2.3 upgrade and how to force an Upgrade on a search node

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



In the Upgrade utility log (/data01/tmp/patch/logs/upgrade-utility.log), the following errors can be seen:

2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsUpgradeOperations: Failed to upgrade DPS. responseCode: 1
2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsRetryTask: Failed to upgrade DPS.
com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade Search.
        at com.emc.vcedpa.upgradeutil.configure.dps.DpsUpgradeOperations.upgrade(DpsUpgradeOperations.java:139)
        at com.emc.vcedpa.upgradeutil.configure.dps.DpsRetryTask.run(DpsRetryTask.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

In dps_upgrade.log, we see the following error:

15/03/19 23:23:11 check_status_on_master() Upgrade is still in progress. This may take some time. Please wait
15/03/19 23:23:11 check_status_on_master() IDPA proceeding to check the version of Search Node.
15/03/19 23:24:11 run_cmd_dps_output() Cmd:ls -l /etc/puppet/status/
15/03/19 23:24:11 run_cmd_dps_output() total 0
15/03/19 23:24:11 run_cmd_dps_output() Status: 0
15/03/19 23:24:12 run_cmd_dps_output() Cmd:cat /usr/local/search/version
15/03/19 23:24:12 run_cmd_dps_output() 18.0.0.1065
15/03/19 23:24:12 run_cmd_dps_output() Status: 0
15/03/19 23:24:12 check_search_version_on_master() 18.0.0.1065
15/03/19 23:24:12 check_search_version_on_master() Could not verify Search version

The issue is with the puppet service on Search not being able to run upgrade.sh script and it fails with the following error.
The puppet_agent.log in the IDPA log bundle can be verified to confirm the failure:

2018-10-24T17:14:39,403|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 45)|Current version is empty!!! Stop upgrade and wait for fixing......

2019-03-15T20:27:05,813|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.
2019-03-15T20:32:05,824|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.

2019-03-16T03:18:12,344|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!
2019-03-16T03:23:12,361|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!


Resolution

1. Login to the ACM GUI.

2. Click on 'Retry' for the DPS (Search) component.

3. As soon as the rollback is done for search, login to the Search Master node.

4. Verify the elasticsearch service is running using following command:
service elasticsearch status

If the service is stopped, restart the service.

service elasticsearch start

5. Verify the puppet and puppetmaster service is running.

service puppetmaster status

service puppet status


6. Force the puppet agent to perform the upgrade on Search.

puppet agent -t

Note: This step needs to be done fairly quickly after restarting the Search Upgrade. We will get around 3-4 minutes until the Upgrade times out. 

7. Monitor the Search Upgrade in the ACM GUI for completion.

Affected Products

Integrated Data Protection Appliance Family

Products

PowerProtect Data Protection Software, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance Software
Article Properties
Article Number: 000168413
Article Type: Solution
Last Modified: 27 May 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.