等待第 6 代 RPA 从 ISO 升级启动时，部署升级失败 —“WARN - Apply Failed. - Connection failed.

版本：3

文章类型：中断修复

目标受众：级别 30 = 客户

上次发布时间：2019 年 5 月 22 日（星期三），04:27:20 GMT

摘要：

通过第 6 代 RPA 升级 RecoverPoint 群集时，Deployment Manager 可能会失败，并显示错误“WARN - Apply Failed. - Connection failed. Verify the RPA is reachable, make sure the login credentials are correct, check user status and try again.”（警告 — 应用失败。— 连接失败。验证 RPA 是否可访问，确保登录凭据正确，检查用户状态并重试。）

问题：

通过第 6 代 RPA 升级 RecoverPoint 群集时，第 6 代 RPA 上的操作可能会失败，并显示错误：
WARN  - Apply Failed. - Connection failed. Verify the RPA is reachable, make sure the login credentials are correct, check user status and try again.（警告 — 应用失败。— 连接失败。验证 RPA 是否可访问，确保登录凭据正确，检查用户状态并重试。）

ui.log (DM Folder\logs\ui.log) 中的错误：
2019-05-18 04:56:23,429 [ApplyNDUUPdaterThread] (ApplyNDUSummaryPage.java:419) INFO  - setProcessStarted[true] RUNNING
2019-05-18 04:56:23,437 [main] (ApplyNDUSummaryPage.java:454) WARN  - Apply Failed. - Connection failed. Verify the RPA is reachable, make sure the login credentials are correct, check user status and try again.
2019-05-18 04:56:29,430 [ApplyNDUUPdaterThread] (ApplyNDUSummaryPage.java:419) INFO  - setProcessStarted[false] FAILED_ERROR

dm.log (DM Folder\logs\dm.log) 中的错误：
2019-05-18 03:46:28,706 [pool-1-thread-13] (NDUApplyCommand.java:1090) DEBUG - Finished ISO upgrading rpa: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]
2019-05-18 03:46:28,706 [pool-1-thread-13] (GeneralUtils.java:115) DEBUG - Sleeping for: Waiting for rpas to go down: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN], time to sleep in millis is: 300000
2019-05-18 03:51:28,719 [pool-1-thread-13] (BaseDMCommand.java:198) DEBUG - Command:VerifyRPAsAreUpCommand Started
2019-05-18 03:51:28,719 [pool-1-thread-13] (VerifyRPAsAreUpCommand.java:37) DEBUG - Verifing rpas are up: [RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]], timeoutInSecs=900
2019-05-18 03:51:28,720 [pool-1-thread-21] (BaseDMCommand.java:198) DEBUG - Command:VerifyBoxUpLogicCommand Started
2019-05-18 03:51:28,721 [pool-1-thread-21] (BaseDMCommand.java:198) DEBUG - Command:AuthenticateCommand Started
2019-05-18 03:51:28,721 [pool-1-thread-21] (BaseInstallationServerAdapter.java:75) DEBUG - Connecting to rpa: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]
2019-05-18 03:51:50,846 [pool-1-thread-21] (BaseInstallationServerAdapter.java:222) INFO  - rpa: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]
 *** has no alternative IS credentials.
2019-05-18 03:51:50,847 [pool-1-thread-21] (BaseInstallationServerAdapter.java:216) ERROR - rpa: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]
 *** no alternative IPs, cannot retry operation: VALIDATE_PASSWORD, params={}
com.kashya.installation.exception.InstallationServiceException: rpa: RPA [ip=XXX.XXX.XXX.XXX, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=Credentials [user=admin, password=*****], alternativeISCredentails=null, number=4, siteName=Site1, version=5.0.SP1.P2(e.265), boxWWNs=null, rpaState=null, generation=UNKNOWN]
 *** has no alternative IS credentials.
...
2019-05-18 04:06:43,322 [pool-1-thread-21] (ReturnValue.java:51) ERROR - 
2019-05-18 04:06:43,322 [pool-1-thread-21] (BaseDMCommand.java:253) DEBUG - Command VerifyBoxUpLogicCommand FINISHED with returnValue=[ReturnValue [_isSuccess=false, _message=Rpa: XXX.XXX.XXX.XXX is down., _data=null, _throwable=com.emc.dm.exception.InstallationFlowException: Rpa: XXX.XXX.XXX.XXX is down., _failureReason=UNKNOWN]]
2019-05-18 04:06:43,322 [pool-1-thread-21] (BaseDMCommand.java:207) DEBUG - Command:VerifyBoxUpLogicCommand Finished
2019-05-18 04:06:43,323 [pool-1-thread-13] (BaseDMCommand.java:152) ERROR - VerifyRPAsAreUpCommand FAILED
...
2019-05-18 04:06:43,324 [pool-1-thread-13] (ReturnValue.java:51) ERROR - Found some RPAs that are down. Their IP addresses are: 
2019-05-18 04:06:43,324 [pool-1-thread-13] (BaseDMCommand.java:253) DEBUG - Command VerifyRPAsAreUpCommand FINISHED with returnValue=[ReturnValue [_isSuccess=false, _message=Found some RPAs that are down. Their IP addresses are: , _data=null, _throwable=com.emc.dm.exception.InstallationFlowException: Rpa: XXX.XXX.XXX.XXX is down., _failureReason=UNKNOWN]]
2019-05-18 04:06:43,324 [pool-1-thread-13] (BaseDMCommand.java:207) DEBUG - Command:VerifyRPAsAreUpCommand Finished
2019-05-18 04:06:43,325 [pool-1-thread-13] (DMUtils.java:51) ERROR - Got a failed return value for: Verifing rpas are up after ISO upgrade, return value is: ReturnValue [_isSuccess=false, _message=Found some RPAs that are down. Their IP addresses are: , _data=null, _throwable=com.emc.dm.exception.InstallationFlowException: Rpa: XXX.XXX.XXX.XXX is down., _failureReason=UNKNOWN]
com.emc.dm.exception.InstallationFlowException: Rpa: XXX.XXX.XXX.XXX is down.
    at com.emc.dm.commands.infra.Commands.throwFirstRVExceptionIfThereAreAny(Commands.java:53)
    at com.emc.dm.commands.infra.Commands.invokeAllAndVerifySuccess(Commands.java:131)
    at com.emc.dm.commands.infra.BaseDMCommand.invokeAllAndVerifySuccess(BaseDMCommand.java:101)
    at com.emc.dm.commands.concrete.verify.VerifyRPAsAreUpCommand.inner(VerifyRPAsAreUpCommand.java:48)
    at com.emc.dm.commands.infra.BaseDMCommand.execute(BaseDMCommand.java:41)
    at com.emc.dm.commands.concrete.status.NDUApplyCommand.runRPAPhaseISOUpgradeStep(NDUApplyCommand.java:459)
    at com.emc.dm.commands.concrete.status.NDUApplyCommand.runRPAPhase(NDUApplyCommand.java:101)
    at com.emc.dm.commands.concrete.status.NDUApplyCommand.inerExecuteAction(NDUApplyCommand.java:73)
    at com.emc.dm.commands.concrete.status.BaseInstallationActionWithStatusLogicCommand.innerWithoutReturnValue(BaseInstallationActionWithStatusLogicCommand.java:37)
    at com.emc.dm.commands.concrete.status.BaseStatusCommand.inner(BaseStatusCommand.java:50)
    at com.emc.dm.commands.infra.BaseDMCommand.execute(BaseDMCommand.java:41)
    at com.emc.dm.commands.concrete.status.ApplyCommand.innerWithoutReturnValue(ApplyCommand.java:48)
    at com.emc.dm.commands.concrete.status.BaseStatusCommand.inner(BaseStatusCommand.java:50)
    at com.emc.dm.commands.infra.BaseDMCommand.execute(BaseDMCommand.java:41)
    at com.emc.dm.commands.infra.BaseDMCommand.call(BaseDMCommand.java:161)
    at com.emc.dm.commands.infra.BaseDMCommand.call(BaseDMCommand.java:32)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.emc.dm.exception.InstallationFlowException: Rpa: XXX.XXX.XXX.XXX is down.
    at com.emc.dm.actions.VerifyBoxUpLogicCommand.inner(VerifyBoxUpLogicCommand.java:40)
    ... 7 more
Caused by: com.emc.recoverpoint.utils.state_observer.StateException: State wasn't reached.
    at com.emc.recoverpoint.utils.state_observer.StateObserver.waitUntilStateIsReached(Predicate.scala:22)
    at com.emc.dm.actions.VerifyBoxUpLogicCommand.inner(VerifyBoxUpLogicCommand.java:38)
    ... 7 more

原因：

对于 RPA 完成 ISO 升级（包括启动）所需的时间，Deployment Manager 出现 2 次超时：
等待 rpa 关闭 — time to sleep in millis is: 300000
验证 rpa 已启动 — timeoutInSecs=900

因此，组合 DM 为 RPA 重新启动分配 20 分钟。
第 6 代 RPA 可能需要略微更长的时间来完成 ISO 升级 — 25 分钟左右

变更：

使用 Deployment Manager，通过第 6 代 RPA 升级 RecoverPoint 群集

解决方法：

解决方法：

等待 DM 出现故障 — 验证 RPA 应答 ping 并允许 DM 重试操作

解决方法：

启动 Deployment Manager 进行升级之前，更改 DM Folder\conf\dm.properties 上的参数
自动升级后，包装从重新启动中启动的超时时间（以秒为单位）
BOX_UP_AFTER_AUTO_UPGRADE_TIMEOUT_IN_SECS = 900

更改为
自动升级后，包装从重新启动中启动的超时时间（以秒为单位）
BOX_UP_AFTER_AUTO_UPGRADE_TIMEOUT_IN_SECS = 1800

主要产品：

RecoverPoint

产品：

RecoverPoint、RecoverPoint CL、RecoverPoint EX、RecoverPoint SE、RecoverPoint Gen6 Server