PowerProtect Data Protection, IDPA: Rapid Upgrade ChecKer Utility Shows Failure

2

POST /versions

Failed to query FW version because PTAgent is not running (or PTA and iSM/iDRAC are not available).

9001

Failed to connect to PTAgent.

Failed to connect to PTAgent. Check PTAgent status or IP connection.

3

POST /versions

Failed to query the FW version because iSM is not running.

Note: When iSM is stopped, it may take up to 3 minutes for PTAgent to process some REST requests, such as POST /host/swinventory and POST /host/lc.

9002

Failed to query SW inventory from PTAgent (HttpStatus.SERVICE_UNAVAILABLE).

Failed to query SW inventory from PTAgent (HttpStatus.SERVICE_UNAVAILABLE). Check iSM/iDRAC status.

4

POST /versions

Failed to query FW version because iDRAC is not ready.

9003

Failed to query SW inventory from PTAgent (HttpStatus.BAD_GATEWAY).

Failed to query SW inventory from PTAgent (HttpStatus.BAD_GATEWAY). Check iDRAC/iSM status.

5

POST /precheck

Failed to query FW version because PTAgent is not running (or PTA and iSM/iDRAC are not available).

9004

Failed to connect to PTAgent.

Failed to connect to PTAgent. Check PTAgent status or IP connection.

6

POST /precheck

FW precheck failed because iSM is not running.

Note: When iSM is stopped, it may take up to 3 minutes for PTAgent to process some REST requests, such as POST /host/swinventory and POST /host/lc.

9005

Failed to query SW inventory from PTAgent (HttpStatus.SERVICE_UNAVAILABLE).

Failed to query SW inventory from PTAgent (HttpStatus.SERVICE_UNAVAILABLE). Check iSM/iDRAC status.

7

POST /precheck

FW precheck failed because iDRAC is not ready.

9006

Failed to query SW inventory from PTAgent (HttpStatus.BAD_GATEWAY).

Failed to query SW inventory from PTAgent (HttpStatus.BAD_GATEWAY). Check iDRAC/iSM status.

8

POST/readinesscheck

ReadinessCheck failed because PTAgent is not running (or PTA and iSM/iDRAC are not available).

N/A

Failed to connect to PTAgent.

Failed to connect to PTAgent. Check PTAgent status or IP connection.

9

POST /readinesscheck

ReadinessCheck failed because iSM is not running (or iSM and iDRAC are not available).

N/A

Failed to query host summary from PTAgent (HttpStatus.SERVICE_UNAVAILABLE).

Failed to query host summary from PTAgent (HttpStatus.SERVICE_UNAVAILABLE). Check iSM/iDRAC status."

Failed to query host summary from Node Event Service (HttpStatus.SERVICE_UNAVAILABLE). Check iSM/iDRAC status."

10

POST /readinesscheck

ReadinessCheck failed because iDRAC is not ready.

N/A

Failed to query host summary from PTAgent (HttpStatus.BAD_GATEWAY).

Failed to query host summary from PTAgent (HttpStatus.BAD_GATEWAY). Check iDRAC/iSM status.

Failed to query host summary from Node Event Service (HttpStatus.BAD_GATEWAY). Check iDRAC/iSM status.

11

POST /readinesscheck

ReadinessCheck failed because iDRAC is in recovery mode.

N/A

Lifecycle Controller is in recovery mode.

Lifecycle Controller is in recovery mode. Clear the recovery mode before FW update.

12

POST /readinesscheck

ReadinessCheck failed because there are some pending jobs in the iDRAC job queue.

N/A

There are some pending jobs in the iDRAC job queue.

There are some pending jobs in the iDRAC job queue. Clear iDRAC job queue before FW update

13

POST /update

GET /activities/{id}
(> IDPA 2.4 only)

FW update Job is stuck at 0% download which causes subsequent tasks to fail.

9012

Failed to unpack firmware payload.

Failed to unpack firmware payload. Check if the iDRAC LC job queue is clear.

14

POST /update

GET /activities/{id}

Failed to update FW because PTAgent is not running.

9013

Failed to connect to PTAgent.

Failed to connect to PTAgent. Check PTAgent status or IP connection.

15

POST /update

GET /activities/{id}
(> IDPA 2.4 only)

Failed to update FW because iSM is not running.

Note: When iSM is stopped, it may take up to 3 minutes for PTAgent to process some REST requests. It might return a timeout error in this case.

9014

Failed to process firmware payload with PTAgent (HttpStatus.SERVICE_UNAVAILABLE).

Failed to process firmware payload with PTAgent (HttpStatus.SERVICE_UNAVAILABLE). Check iSM/iDRAC status.

16

POST /update

GET /activities/{id}

Failed to update FW because iDRAC is not ready.

9015

Failed to process firmware payload with PTAgent (HttpStatus.BAD_GATEWAY).

Failed to process firmware payload with PTAgent (HttpStatus.BAD_GATEWAY). Check iDRAC/iSM status.

17

GET /precheck

FW precheck failed due to an error "No FW profile found."

9017

No FW profile found for <IDPA model>.

No FW profile found <IDPA model>. Ensure that the correct ID Module is installed.

POST /postupdate

GET /activities/{id}

POST /postupdate

GET /activities/{id}

POST /preupdate

GET /activities/{id}

POST /preupdate

GET /activities/{id}

Failed to perform pre-requisite tasks due to an internal error.

POST /update

GET /activities/{id}

FW payload file is not found. Add the firmware payload path to the request body and then retry the FW update API.

POST /update

GET /activities/{id}

Failed to update firmware due to an internal error. Check PTAgent/iSM/iDRAC status and review logs for details.

POST /preupdate

GET /activities/{id}

POST /postupdate

GET /activities/{id}

Failed to perform postupdate tasks because PTAgent is not running.

POST /postupdate

GET /activities/{id}

Failed to perform postupdate tasks due to an internal error.

Timeout while waiting for the REST API response

POST /postupdate

GET /activities/{id}

ESXi host failed to exit maintenance mode. Check ESXi host status and ensure vSAN is in a healthy state.

POST /postupdate

GET /activities/{id}

Timeout while waiting for host up and running
Check ESXi host status and ensure vSAN is in a healthy state.

POST /update

GET /activities/{id}

The maximum wait time for system reset is exceeded. Check iDRAC and PTAgent status.

Failed to verify if the current firmware is valid. 

POST /update

GET /activities/{id}

POST /readinesscheck

ReadinessCheck failed because PTAgent is not running (or PTA service is not available).

N/A

PTAgent is not available and active now.

PTAgent is not available and active now. Check PTAgent Service status or IP connection.

POST /readinesscheck

ReadinessCheck failed because iSM is not running (or service is degraded or disabled).

N/A

The cached response with Node Event Service is disabled.

Node Event Service is in a degraded state, iDRAC Service Module is not available or active now.

Multiple known issues:

See article 219231 Integrated Data Protection Appliance: The Cached Response With Node Event Service Is Disabled. Node Event Service Is in Degraded Due to a Duplicate Route 169.254.0.1 


See article 197174 PowerProtect Data Protection Appliances, IDPA: PowerProtect Data Protection Rapid Upgrade Checker Reported a Firmware Upgrade precheck failure

See article 219233 Integrated Data Protection Appliance: The Cached Response Degraded ESXi /tmp Folder Full

1. SSH to ESXi Host and run following command to correct the PTAgent rest_ip configuration:

• /opt/dell/DellPTAgent/tools/pta_cfg set rest_ip=https://<host_internal_IP>:8086

• /etc/init.d/DellPTAgent status

• /etc/init.d/DellPTAgent start

• /etc/init.d/DellPTAgent restart

2. To avoid further precheck failures, login to all ESXi Hosts and check PTAgent service status manually. If any Dell PTAgent service is down, start it manually.

1. Log in to an ESXi Host in which iSM issue is observed.
2. Run command:
   1. /etc/init.d/dcism-netmon-watchdog status

1. 2. /etc/init.d/dcism-netmon-watchdog start 

1. SSH into iDRAC from ACM CLI using command "ssh root@<iDRAC-IP-Address>" (In order to fetch the iDRAC IP Address, following command can be used: enum_instances OMC_IPMIIPProtocolEndpoint root/cimv2 | grep IPv4Address).
2. Run the command racadm jobqueue delete -i JID_CLEARALL_FORCE
3. Wait 5 minutes for iDRAC to settle down.

Example:

racadm>>racadm jobqueue delete -i JID_CLEARALL_FORCE
RAC1032: JID_CLEARALL_FORCE job was canceled by the user.
racadm>> 

If iDRAC UI shows that iSM is "Not running (TLS error)," apply the following workaround:

1. Log in to an ESXi Host in which iSM issue is observed.
2. Run commands:
   1. /etc/init.d/dcism-netmon-watchdog stop
   2. /etc/init.d/dcism-netmon-watchdog start install

1. Log in to an ESXi Host in which iSM issue is observed.
2. Run command:
   1. /etc/init.d/dcism-netmon-watchdog status 

2. 2. /etc/init.d/dcism-netmon-watchdog stop
3. /etc/init.d/dcism-netmon-watchdog start

It may take ~5 minutes to restart.

4. Reset iDRAC
   1. SSH into iDRAC from ACM CLI using command ssh root@<iDRAC-IP-Address> (In order to fetch the iDRAC IP Address, following command can be used: enum_instances OMC_IPMIIPProtocolEndpoint root/cimv2 | grep IPv4Address
   2. Run command: racadm racreset soft

1. SSH to ACM
2. Verify if dpatools version 2.3.0 or higher is already installed.

• To check this run: rpm -qa | grep dpatools

3. Verify if firmware bundle "IDPA-10.308-10.308.tar.gz" is already available at "/usr/local/dpatools/bin/payload."
4. Run command: dpacli -fwupdate /usr/local/dpatools/bin/payload/IDPA-10.308-10.308.tar.gz -skipReboot

After this command finishes, iDRAC version 3.36.103.36 gets installed on nodes.

5. Click the Revalidate button in the Upgrade UI to run the upgrade prechecks again.

One of the possible reasons of this failure is that some client task scripts are missing on ESXi hosts.
This may happen if the ESXi host is not power cycled during the reimage process.
To check and correct this, follow the steps below:

1. SSH to ESXi hosts
2. Write down the list of files present in the '/scratch/dell/extern' folder.
3. SSH to ACM
4. Write down the list of files present in the '/usr/local/dpatools/bin/clienttask' folder.
5. If not all files from ACM's folder are present on ESXi's folder, SCP the files to ESXi hosts.
6. Retry upgrade from Upgrade UI.
7. If the issue persists, contact support.

Another possible reason of this failure is the dpatools-service is not upgraded properly.

To check the dpatools-service version, follow the steps below:

1. SSH to ACM
2. Verify if the dpatools-service target version is already installed 

              rpm -qa | grep dpatools-service

 If the failure happens during ESXi and Firmware, apply latest DPATools, follow the steps below to work around the issue:

1.  Run DPATools CLI command to update firmware on all hosts: 

dpacli -fwworkflow /usr/local/dpatools/bin/payload/IDPA-<version>-<version>.tar.gz

2. Run Upgrade Sync from Upgrade UI.
3. If the issue persists contact support.

One of the possible reasons of this failure is there is a problem with PM1735 NVMe FW or PCIe riser.

By default, PTAgent does automatic device-scan and cache the storage device properties.
PTAgent failed to find component 'PhysicalDisk' in hwInventory because it has trouble to inquire data from the NVMe PM1735 card in slot #4.

The following workarounds can be applied to resolve this issue:

1. Upgrade NVMe/PCIe 1735 firmware to 2.3.0 (For NVMe 1735 drive, V2.0.2 or later is acceptable).
   1. Download 14G FW 2021-December-Block from the Dell Support Product Page site: 
   2. Extract the package, and the contents contain the 1735 FW Express-Flash-PCIe-SSD_Firmware_RP8RC_WN64_2.3.0_A03.EXE.
   3. Enable service mode on the appliance by running the following command on ACM:  dpacli -servicemode
   4. Update NVMe 1735 FW to 2.3.0 using iDRAC manually. 
   5. Powercycle the appliance for a firmware update to be committed. 

2. If the steps above do not work and the issue persists, attempt a power off recycle sequence:
   1. On ACM, Enable service mode on the appliance by running the following command on ACM: dpacli -servicemode
   2. Power down
   3. Unplug power cable
   4. Hold down the power button for 10 seconds.
   5. Plug in power cable
   6. Power up
3. If steps 1 and 2 listed above do not resolve the issue, it could be an issue with slot or riser.
   1. Replace the PCIe riser.