Recalls are failing, restarting the callback daemon temporarily resolves the issue.
Summary: Recalls are failing, and a restart of the atmos callback daemon resolves the issue.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Multiple errors in the acd_debug.log
Example errors:
Apr 05 16:34:01 AtmosCallbackDae(2918) [28-89-4-360]: (ERROR) handler thread id was unable to process request threadid:140607314130688 url:/fmroot/Wasabi/creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43?comp-fsize=53630995%enc-iv=bTya_kizwHnTObr5C8fiKw==%enc-key=ipW2aReOv2mlc85XWbATkPPQF5ZM8troKAaM4iL+V2bPoW9DfKA683IGLIwDJ83o this error required clientsocket to be closed
Apr 05 16:34:01 AtmosCallbackDae(2918) [28-89-4-72]: (ERROR) AtmosCallBackManager::ProcessHTTPRequestWriteResponseToRequest FAILS threadid=140607314130688 socket=69
Apr 05 16:34:01 AtmosCallbackDae(2917) [28-89-4-280]: (ERROR) AtmosRequestHandler::writeResponse socket write returns error recall file=/fmroot/Wasabi/creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43?comp-fsize=53630995%enc-iv=bTya_kizwHnTObr5C8fiKw==%enc-key=ipW2aReOv2mlc85XWbATkPPQF5ZM8troKAaM4iL+V2bPoW9DfKA683IGLIwDJ83o CloudName=Wasabi ObjectID=creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43 threadid=140607316231936
Example errors:
Apr 05 16:34:01 AtmosCallbackDae(2918) [28-89-4-360]: (ERROR) handler thread id was unable to process request threadid:140607314130688 url:/fmroot/Wasabi/creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43?comp-fsize=53630995%enc-iv=bTya_kizwHnTObr5C8fiKw==%enc-key=ipW2aReOv2mlc85XWbATkPPQF5ZM8troKAaM4iL+V2bPoW9DfKA683IGLIwDJ83o this error required clientsocket to be closed
Apr 05 16:34:01 AtmosCallbackDae(2918) [28-89-4-72]: (ERROR) AtmosCallBackManager::ProcessHTTPRequestWriteResponseToRequest FAILS threadid=140607314130688 socket=69
Apr 05 16:34:01 AtmosCallbackDae(2917) [28-89-4-280]: (ERROR) AtmosRequestHandler::writeResponse socket write returns error recall file=/fmroot/Wasabi/creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43?comp-fsize=53630995%enc-iv=bTya_kizwHnTObr5C8fiKw==%enc-key=ipW2aReOv2mlc85XWbATkPPQF5ZM8troKAaM4iL+V2bPoW9DfKA683IGLIwDJ83o CloudName=Wasabi ObjectID=creeled-americas-archive+cta_10.22.10.36_1680231855_5842_43 threadid=140607316231936
Information in the acd_debug log reports:
Cause
The number of recall requests coming to CTA has reached 1001.
CTA by default accepts max 1000 recall requests coming to it at any point of time.
Anything more than that it cannot accept, further recall requests fail to be served.
CTA by default accepts max 1000 recall requests coming to it at any point of time.
Anything more than that it cannot accept, further recall requests fail to be served.
Resolution
Restarting ACD temporarily resolves issue as it frees up all CTA recall threads, which then again get consumed 100% after some time.
This means that there is lot of recall load on CTA or lot of hot/active data has been archived which is being recalled frequently.
By design the CTA is not meant to handle hot data or give performance for high recall count. It is meant to handle cold data.
In order to resolve this. The customer must identify from where this high recalls are being done in their environment, which Filesystems are recalling much and why.
If they have archived data which must be read frequently, then they have to rehydrate them back and reassess their archive policy in CTA such that active data is not archived.
These steps should resolve the current issue permanently.
This means that there is lot of recall load on CTA or lot of hot/active data has been archived which is being recalled frequently.
By design the CTA is not meant to handle hot data or give performance for high recall count. It is meant to handle cold data.
In order to resolve this. The customer must identify from where this high recalls are being done in their environment, which Filesystems are recalling much and why.
If they have archived data which must be read frequently, then they have to rehydrate them back and reassess their archive policy in CTA such that active data is not archived.
These steps should resolve the current issue permanently.
Affected Products
Cloud Tiering Appliance, Cloud Tiering Appliance, Cloud Tiering Appliance Platform, Cloud Tiering Appliance/VEArticle Properties
Article Number: 000213109
Article Type: Solution
Last Modified: 29 Jun 2023
Version: 1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.