Integrated Data Protection Appliance: The Cached Response Degraded ESXi /tmp Folder Full

RPS Precheck Failure:

[ERROR] Firmware pre-upgrade checks failed. [ 10.xx.xxx.xx ]: The cached response with Node Event Service is disabled. Node Event Service is in a degraded state,iDRAC Service Module is not available/active at this time. Check iDRAC Service Module/iDRAC status.

showfru disk
Collecting Data............
Failed to execute DPA CLI command: 503 Service Unavailable
Usage: "showfru -help" for more details.

For other related firmware_readiness checks caught in RUCK, PowerProtect DP Series and IDPA: Rapid Upgrade ChecKer Utility shows failure for "firmware_readiness" check

ESXi /tmp folder full

In the PTA debug log, /scratch/log/pta_debug.log you see the following errors:
 

2023/10/06 15:53:44[UTC] [26119210:194411328] WARN  - RefreshDomainHostNames: Command '/bin/esxcli system hostname get' failed - fork(/bin/esxcli) failed - No space left on device.
2023/10/06 15:55:16[UTC] [26119210:189655872] ERROR - execCmdNewEsxi: Error during command execution. Return value from command output - <255>
2023/10/06 15:55:16[UTC] [26119210:189655872] WARN  - PerformanceMonitor::getMemoryUsed: Unable to retrieve memory performance information. Either the group or the current process name is empty.
2023/10/06 15:56:17[UTC] [26119210:184900416] ERROR - System call terminated abnormally: errno=28
2023/10/06 15:56:17[UTC] [26119210:184900416] ERROR - Following file does not exist : /tmp/3f65d64c-67c0-4e69-ab7c-c9609d7a517f.PTAgent.tmp

To resolve this issue, do the following:

1. Clean /tmp on ESXi
2. Restart both iSM and PTAgent with the below commands:

/etc/init.d/DellPTAgent restart
/etc/init.d/dcism-netmon-watchdog restart

3. Reset iDRAC
4. Additionally, you can restart the sfcbd service with the following command:

/etc/init.d/sfcbd-watchdog restart

5. Check agent/info and firmware/readiness check are both OK now with the following commands:

/opt/dell/DellPTAgent/tools/pta_call get agent/info

DP4400

curl -k -i -H "Content-Type:application/json" -X POST  https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": false, "hostList": [{"hostIP": "192.168.100.101", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'

You will get an output like the following:

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:38:41 GMT

[{"id":814402494717874004,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]acm-4400-crk-3:~


DP5000/DP8000


for i in {1..3}; do curl -k -i -H "Content-Type:application/json" -X POST https://localhost:8039/dpatools/api/v1/firmware/readinesscheck -d '{"idpaVersion": "2.7", "isRack": true, "hostList": [{"hostIP": "192.168.100.10'${i}'", "esxiUser": "root", "esxiPassword": "IDPAPASSWORD"}]}'; done

HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:05 GMT

[{"id":2337625064092707360,"hostIP":"192.168.100.101","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":false,"twoHopIsRequired":false,"invalidFirmwareList":[{"name":"Integrated Remote Access Controller","currentVersion":"3.36.103.36","minimumRequiredVersion":"4.40.10.00"}],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:39:44 GMT

[{"id":1389413431580575643,"hostIP":"192.168.100.102","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 05 Nov 2023 01:40:19 GMT

[{"id":1184184063038749065,"hostIP":"192.168.100.103","result":"SUCCESS","agentIdracDataCached":true,"agentIsOverallReady":true,"agentIsRunning":true,"agentVersionIsOk":true,"idracInRecoveryMode":false,"idracIsAvailable":true,"idracIsReady":true,"idracLcJobQueueIsClear":true,"ismIsEnabled":true,"ismIsReady":true,"firmwareIsValid":true,"twoHopIsRequired":false,"invalidFirmwareList":[],"messages":"The IDPA system is ready for firmware update.","links":[]}]acm-5800-crk:~