RecoverPoint for VMs: Upgrade from 5.3.0.x or 5.3.1.0 may stop responding at 11% or 84%
Summary:
Upgrade from RecoverPoint for VMs 5.3.0.x or 5.3.1.0 may fail with possible error: "Error: Upgrade had failed. If upgrade keeps failing, contact customer support. vRPA is not up"
Symptoms
There are several different possible symptoms from the same root cause:
Symptom 1:
The Debian package upgrade did not go through even though installation logs say it was successful.
Symptom 2:
Tomcat service did not come up after reboot.
Symptom 3:
One or more of the following messages are seen in clusterLogic logs in RPA1 or RPA2:
ERROR - Failed to connect to vRPA with IP <IP>. com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused)
ERROR - UpgradeRemoteRPATask :Task failed: Upgrading vRPA number <RPA NUMBER>. com.emc.recoverpoint.utils.javainfra.status.tasks.exception.TaskFailedException: vRPA <IP> is not up.
Symptom 4:
The following messages are seen on the UI:
The error seen in UI is "Error: Upgrade had failed. If upgrade keeps failing, contact customer support. vRPA <IP> is not up"
Cause
When the RPA is being upgraded, the Debian packages (.deb files) are installed and a reboot is performed.
The package installation runs in the background which may get aborted by the reboot in the next step of upgrade.
Because the installer is killed before completion, the tomcat8.service may not be available in the systemd directory, resulting in failure to restart tomcat8 and causing the RPA to be unreachable.
Resolution
Preventative workaround:
Login as admin user to each RPA before the upgrade and run the following signed script ([2] Setup -> [8] Advanced options -> [4] Run script
OTBkOWY0ZDA1MjMxODMxNWM2NTJkMjZjOGYyM2E1MWQKdW5saW1pdGVkCm5vdF9yZXN0cmljdGVk ClRoZSBpZCBvZiB0aGUgc2NyaXB0IGlzOlJQLTMxNzE4CkFkZHMgc2xlZXAgdG8gYWxsb3cgdXBn cmFkZSB0byBjb21wbGV0ZQpBc3NpZiBIYWwKVkVSU0lPTj0kKGdyZXAgdF92ZXJzaW9uX2Z1bGwg L2hvbWUva29zL2tib3gvc3JjL2luaXRpYWxpemF0aW9uL3R3ZWFrX3BhcmFtcy90d2Vhay5wYXJh bXMudmVyc2lvbnxncmVwIC1vICJbMS05XS4qWzAtOV0iKQpNQUpPUlZFUlNJT049JChlY2hvICIk VkVSU0lPTiJ8YXdrIC1GICIuIiAne3ByaW50ICQxfScpCk1JTk9SVkVSU0lPTj0kKGVjaG8gIiRW RVJTSU9OInxhd2sgLUYgIi4iICd7cHJpbnQgJDJ9JykKU1BWRVJTSU9OPSQoZWNobyAiJFZFUlNJ T04ifGF3ayAtRiAiLiIgJ3twcmludCAkM30nKQpQQVRDSFZFUlNJT049JChlY2hvICIkVkVSU0lP TiJ8YXdrIC1GICIuIiAne3ByaW50ICQ0fScpCmlmIChbICRNQUpPUlZFUlNJT04gLWVxIDUgXSAm JiBbICRNSU5PUlZFUlNJT04gLWVxIDMgXSAmJiBbICRTUFZFUlNJT04gLWVxIDAgXSkgfHwgKFsg JE1BSk9SVkVSU0lPTiAtZXEgNSBdICYmIFsgJE1JTk9SVkVSU0lPTiAtZXEgMyBdICYmIFsgJFNQ VkVSU0lPTiAtZXEgMSBdICYmIFsgJFBBVENIVkVSU0lPTiAtZXEgMCBdKTsKCXRoZW4KCQlpZiBb IGBncmVwICJzbGVlcCA2MDAiIC9ob21lL2tvcy9rYm94L3NyYy9pbnN0YWxsYXRpb24vZGlzdHJp YnV0aW9uL2lzb19zZXJ2aWNlcy5zaHx3YyAtbGAgLWVxIDAgXTsKCQkJdGhlbgoJCQkJc2VkIC1p ICcxODYgaSBzbGVlcCA2MDAnIC9ob21lL2tvcy9rYm94L3NyYy9pbnN0YWxsYXRpb24vZGlzdHJp YnV0aW9uL2lzb19zZXJ2aWNlcy5zaAoJCQkJc2VkIC1pICJzL3NsZWVwIDYwMC8gICBzbGVlcCA2 MDAvIiAvaG9tZS9rb3Mva2JveC9zcmMvaW5zdGFsbGF0aW9uL2Rpc3RyaWJ1dGlvbi9pc29fc2Vy dmljZXMuc2gKCQkJCWVjaG8gIkNoYW5nZXMgYXBwbGllZCBzdWNjZXNzZnVsbHkgdG8gaXNvX3Nl cnZpY2VzLnNoLiIKCQllbHNlCgkJCWVjaG8gIlZhbHVlIGFscmVhZHkgYWRkZWQuIE5vIGNoYW5n ZXMgbWFkZS4iCgkJZmkKZWxzZQoJZWNobyAiTm8gbmVlZCB0byBydW4gdGhpcyBzY3JpcHQgb24g dGhpcyB2ZXJzaW9uLiBObyBjaGFuZ2VzIG1hZGUuIgpmaQo= #
Resolution:
This issue is resolved in RecoverPoint 5.3 SP1 Patch 1 and later code versions.
Upgrade from 5.3 SP1 P1 and later should prevent the issue from occurring.
Additional Information
For more information, consult Jira defect number RP-31718. Jira access is only available to authorized Customer Service Representatives.