Dell EMC Unity: Unable to Access Unisphere The system is busy. Try again later (User Correctable)
Summary: Dell EMC Unity: Unable to Access Unisphere The system is busy. Try again later (User Correctable)
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
In Unity 4.5.0 and later, the ECOM request handler number was changed to a limited number to avoid out of memory issue when too many request handlers are handling requests. This limit is set on ECOM startup based on available resources (CPU number, free memory size) on the array. This change made it more likely for the problem to be encountered.
Unisphere reports the following message when trying to access Unisphere The system is busy. Try again later. The array is still pingable and accessible via CLI. Upon review of the array via CLI, the Management service reports as disabled.
The following Command can be run to confirm the Management services status.
Command >> svc_restart_service status MGMT
The server.log file shows a failed attempt when the GUI cannot be started:
Log Location: /spx/usr/apache-tomcat/logs/server.log:
Unisphere reports the following message when trying to access Unisphere The system is busy. Try again later. The array is still pingable and accessible via CLI. Upon review of the array via CLI, the Management service reports as disabled.
The following Command can be run to confirm the Management services status.
Command >> svc_restart_service status MGMT
| 19:52:00 service@test spa:~/user# svc_restart_service status MGMT MGMT is DOWN |
The server.log file shows a failed attempt when the GUI cannot be started:
Log Location: /spx/usr/apache-tomcat/logs/server.log:
UnirestExceptionUtil:convertUnirestExceptionIntoError-- [ErrorCode: 0x7d13002] [Message: The system is busy. Try again later. If the problem persists, search for the error code on the support website or product forums, or contact your service provider, if available. (Error Code:0x7d13002)] |
Cause
We have identified an issue with Third-Party Vendor Software which issues connections to the array's Management Service, where the Third-Party Software does not always terminate connections. When the Third-Party Software service is disabled on the hosts on which it is running, we see that the problem is avoided and connections are successfully terminated. We have seen this issue with the Third-Party Software such as ServiceNow.
To identify indications of this issue, we can review the memory.log on the Unity. The Memory.log file reports a periodic snapshot of the thread count. We can see that the thread count increases at various intervals before the issue is experienced.
Log Location: /spx/EMC/C4Core/log/memory.log:
Note: The number of Threads will be the 18th entry after "(ECOM)" is reported.
Example Breakdown, The total number of Threads here is "130" as it is the 18th entry:
Below is the example output showing the number of threads increasing overtime. (Please note that the thread count will be unique for each array)
Log entry shows total thread count is currently at 153 threads:
Log entry about 1 day later shows the total thread count has increased to 161 threads:
To identify indications of this issue, we can review the memory.log on the Unity. The Memory.log file reports a periodic snapshot of the thread count. We can see that the thread count increases at various intervals before the issue is experienced.
Log Location: /spx/EMC/C4Core/log/memory.log:
Note: The number of Threads will be the 18th entry after "(ECOM)" is reported.
Example Breakdown, The total number of Threads here is "130" as it is the 18th entry:
| INDEX | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| (ECOM) | S | 18113 | 14403 | 14403 | 0 | -1 | 1077944576 | 108196161 | 1356451908 | 6646 | 6975 | 806452 | 398756 | 1292712 | 773547 | 20 | 0 | 130 | 0 |
Below is the example output showing the number of threads increasing overtime. (Please note that the thread count will be unique for each array)
Log entry shows total thread count is currently at 153 threads:
| 8601923 Mon Aug 12 23:20:51 18269 (ECOM) S 18268 9054 9054 0 -1 1077944576 65520255 586277653 10129 5401 2695418 483364 566839 333778 20 0 153 0 805147515 867745792 20558 <..> |
Log entry about 1 day later shows the total thread count has increased to 161 threads:
| 8648731 Tue Aug 13 12:21:00 18269 (ECOM) S 18268 9054 9054 0 -1 1077944576 71093367 637402755 11337 6302 2943828 526182 617259 363530 20 0 161 0 805147515 868909056 20858 <..> |
Resolution
As a temporary workaround, access can be restored by 'svc_restart_service MGMT restart'.
Note that this workaround is to regain temporary access to Unisphere, since the issue with the Third-Party Software still exists in the environment, the workaround may have to be periodically re-applied until the issue is resolved on the Third-Party Software.
Command >> svc_restart_service restart MGMT
Note that this workaround is to regain temporary access to Unisphere, since the issue with the Third-Party Software still exists in the environment, the workaround may have to be periodically re-applied until the issue is resolved on the Third-Party Software.
Command >> svc_restart_service restart MGMT
| 19:58:00 service@test spa:~/user# svc_restart_service restart MGMT Found ECOM on spb |
Additional Information
This content is translated in different languages:
| https://downloads.dell.com/TranslatedPDF/AR-SA_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/DE_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/ES_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/ES-XL_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/FR_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/IT_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/JA_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/KO_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/NL_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/PT_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/PT-BR_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/RU_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/SV_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/ZH-CN_537221.pdf |
| https://downloads.dell.com/TranslatedPDF/ZH-TW_537221.pdf |
Affected Products
Dell EMC Unity FamilyProducts
Dell Unity 300, Dell EMC Unity 300F, Dell EMC Unity 350F, Dell EMC Unity XT 380, Dell EMC Unity XT 380F, Dell EMC Unity 400, Dell EMC Unity 400F, Dell EMC Unity 450F, Dell EMC Unity XT 480, Dell EMC Unity XT 480F, Dell EMC Unity 500
, Dell EMC Unity 500F, Dell EMC Unity 600, Dell EMC Unity 600F, Dell EMC Unity 650F, Dell EMC Unity XT 680, Dell EMC Unity XT 880, Dell EMC Unity XT 880F, Dell EMC Unity Family |Dell EMC Unity All Flash, Dell EMC Unity Family, Dell EMC Unity Hybrid
...
Article Properties
Article Number: 000056109
Article Type: Solution
Last Modified: 09 Oct 2024
Version: 4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.