VxRail: ESXCLI Commands Are Stuck

Summary: When running esxcli commands, the host stops responding and shows no output.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms

Trying to run any esxcli commands, for example:

esxcli network ip connection list |grep 9090

The command is stuck.

The command is stuck 

Hostd.log:

2024-08-29T13:03:10.481Z warning hostd[2103711] [Originator@6876 sub=RateLimiter opID=b29c9240 user=esximgmt] Request rejected, 785 similar already pending, expected to take over 21745461 minutes (0 27701 0 1029 1029).
2024-08-29T13:03:10.481Z info hostd[2103711] [Originator@6876 sub=Solo.Vmomi opID=b29c9240 user=esximgmt] Activation finished; <<52d18357-9bfe-7338-41b5-577464ffa3dc, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 19359'>>, ha-cli-handler-system-version, vim.EsxCLI.system.version.get>
2024-08-29T13:03:10.481Z verbose hostd[2103711] [Originator@6876 sub=Solo.Vmomi opID=b29c9240 user=esximgmt] Arg alternate:
--> (null)
2024-08-29T13:03:10.481Z info hostd[2103711] [Originator@6876 sub=Solo.Vmomi opID=b29c9240 user=esximgmt] Throw vmodl.fault.SystemError
2024-08-29T13:03:10.481Z info hostd[2103711] [Originator@6876 sub=Solo.Vmomi opID=b29c9240 user=esximgmt] Result:
--> (vmodl.fault.SystemError) {
-->    reason = "Too many outstanding operations",

Cause

This appears to be related to pending requests, it is not yet clear what the root cause is.

Resolution

Workaround:

1. Put the host in Maintenance Mode (MM).
2. Try to restart the management services:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

3. Try to run the same ESXCLI command again and check the output.
4. If it works, take the host out of MM, if not, reboot the host.

Article Properties
Article Number: 000228159
Article Type: Solution
Last Modified: 20 Jan 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.