IDPA: dpacli -servicemode command fails on DP4400 appliance with error "[akka://DPACLI-System/user/DPARootActor] null"

When the dpacli command is issued to put the appliance in service mode, the command fails with following error:    

xxxxxxacm:/usr/local/dpatools/logs # dpacli -servicemode
[ERROR] [11/04/2019 13:45:36.898] [DPACLI-System-akka.actor.default-dispatcher-2] [akka://DPACLI-System/user/DPARootActor] null
java.lang.NullPointerException
        at com.emc.dpa.dputils.persistence.ACMConfigUtils.getHostIPList(ACMConfigUtils.java:747)
        at com.emc.dpa.dputils.persistence.ACMConfigUtils.getESXiHostInfoList(ACMConfigUtils.java:714)
        at com.emc.dpa.common.akka.DPARootActor.getOpMessageList(DPARootActor.java:503)
        at com.emc.dpa.common.akka.DPARootActor.receiveTaskOpMessage(DPARootActor.java:126)
        at com.emc.dpa.common.akka.DPARootActor.receiveDPAOpMessage(DPARootActor.java:109)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
        at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
        at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:180)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
        at akka.actor.ActorCell.invoke(ActorCell.scala:557)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
        at akka.dispatch.Mailbox.run(Mailbox.scala:225)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

Use the dpacli.log to verify review of the error for this issue (location: /usr/local/dpatools/logs/dpacli.log).

The following error is seen in the dpacli.log:    

[ERROR] [11/04/2019 13:45:36.898] [DPACLI-System-akka.actor.default-dispatcher-2] [akka://DPACLI-System/user/DPARootActor] null
java.lang.NullPointerException
        at com.emc.dpa.dputils.persistence.ACMConfigUtils.getHostIPList(ACMConfigUtils.java:747)
        at com.emc.dpa.dputils.persistence.ACMConfigUtils.getESXiHostInfoList(ACMConfigUtils.java:714)
        at com.emc.dpa.common.akka.DPARootActor.getOpMessageList(DPARootActor.java:503)
        at com.emc.dpa.common.akka.DPARootActor.receiveTaskOpMessage(DPARootActor.java:126)
        at com.emc.dpa.common.akka.DPARootActor.receiveDPAOpMessage(DPARootActor.java:109)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
        at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
        at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:180)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
        at akka.actor.ActorCell.invoke(ActorCell.scala:557)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
        at akka.dispatch.Mailbox.run(Mailbox.scala:225)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

-11-04T16:03:14.617Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-4] [][][][][] com.emc.dpa.dputils.persistence.ACMConfigUtils - ACM configuration file /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml doesn't exist:
2019-11-04T16:03:17.792Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.vmware.vim25.ws.WSClient - Exception caught while invoking method: RetrieveServiceContent
2019-11-04T16:03:17.793Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.emc.dpa.dputils.vi.ViJavaUtils - Error while creating ViJava service instance {}
2019-11-04T16:03:17.828Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.emc.dpa.dputils.vi.ViJavaUtils - Exception occurred while creating ViJava service instance with persisted vCenter parameters {}
2019-11-04T16:03:17.828Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.emc.dpa.dputils.DPApplianceUtils - Unable to connect to VCSA, so cannot shutdown
2019-11-04T16:03:17.828Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.emc.dpa.dputils.DataDomainUtils - Exception occured while test connection of DD: {}
2019-11-04T16:06:04.692Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-6] [][][][][] com.emc.dpa.dputils.persistence.ACMConfigUtils - ACM configuration file /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml doesn't exist:
2019-11-04T16:06:24.920Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-5] [][][][][] com.emc.dpa.dputils.persistence.ACMConfigUtils - ACM configuration file /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml doesn't exist:
2019-11-04T16:06:44.350Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-2] [][][][][] com.emc.dpa.dputils.persistence.ACMConfigUtils - ACM configuration file /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml doesn't exist:
2019-11-04T16:22:52.753Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-4] [][][][][] com.emc.dpa.dputils.persistence.ACMConfigUtils - ACM configuration file /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml doesn't exist:
2019-11-04T16:22:53.114Z ERROR [] [DPACLI-System-akka.actor.default-dispatcher-3] [][][][][] com.emc.dpa.dputils.DataDomainUtils - Exception occured while test connection of DD: {}

Review the DataProtectionConfiguration.xml file for more details. A problematic DataProtectionConfiguration.xml" file may look like the following:    

 cat /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml

Example:    

acmhostname:# cat /usr/local/dataprotection/var/configmgr/server_data/config/DataProtectionConfiguration.xml
Failed to retrieve information about all the IDPA components required to generate PDF file. Please fix all the errors reported on dashboard and retry the
PDF generation operation. Error details: Unable to get vCenter version.

Note: Error details might be different based on different scenarios and issues. 

This confirms that the "DataProtectionConfiguration.xml" file was not populated properly due to ACM not being able to connect with one or more components. 

The "DataProtectionConfiguration.xml" in location "/usr/local/dataprotection/var/configmgr/server_data/config/" is not populated correctly on the ACM. DPATOOLS read this file to gather the component information and then issues test connection query to each component.

If this file is not populated correctly, it causes dpacli -servicemode command to fail with null exception.

1. Fix the errors present on "DataProtectionConfiguration.xml" file. Errors are seen in server.log and are fixed as per the scenario.

Note: Follow resolution provided in notes section if the exact error seen with the file is "Unable to fetch vCenter version"

2. Once the errors are fixed, move/rename the existing DataProtectionConfiguration.xml and DataProtectionConfiguration.pdf files.

• cd /usr/local/dataprotection/var/configmgr/server_data/config/

• mv DataProtectionConfiguration.xml x-DataProtectionConfiguration.xml.old

• mv DataProtectionConfiguration.pdf x-DataProtectionConfiguration.pdf.old

3. Proceed to generate a new IDPA configuration PDF from the ACM GUI.

4. Verify the file is populated correctly.

5. Retry the "dpacli -servicemode" command to verify if the appliance goes to service mode now and proceed with Firmware Update. 

Regarding the error, "Error details: Unable to get vCenter version" seen on "DataProtectionConfiguration.xml" file, the following KB article 538937 can be followed: IDPA: vCenter shows exclamation mark next to it on the ACM UI and Error "Unable to get vCenter version" can be seen on the Virtualization dashboard