ECS: DD Cloud unit disconnected due to http 403 errors - Access Denied
Summary: Cloud unit disconnected due to http 403 errors - Access Denied issue reporting on Data Domain
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Cloud unit gets disconnected due to http 403 errors with Access Denied issue.
The following errors would be reporting on DD.
On ECS, kpi.sh for the affected bucket shows 403 errors.
After tracing the 403 errors, the following time skew alert would be seen in logs.
The following errors would be reporting on DD.
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_curl_write_cb:1680 - Request failed with httpcode:403
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_request_convert_httpcode_to_err:1554 - HTTP operation returned code:403, request error:Access Denied [5075], uri:https://emcecs.xxxxxxx.xx.xx:9021/?endpoint, date:Tue, 21 Mar 2023 03:22:01 GMT, bytes_sent:0, bytes:rcvd:0
03/21 07:22:01.838 (tid 0x6991b0): ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
03/21 07:22:01 ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
04/03 18:46:49.710 (tid 0x699060): ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
04/03 18:46:49 ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
On ECS, kpi.sh for the affected bucket shows 403 errors.
Extracting RequestLog data: DONE
All Requests 500 Errors
Node GETs PUTs POSTs DELETEs HEADs Total GETs PUTs POSTs DELETEs HEADs Total
10.xx.xx.001 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.002 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.003 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.004 1 0 0 0 0 1 0 0 0 0 0 0
10.xx.xx.005 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.006 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.007 0 1 0 0 0 1 0 0 0 0 0 0
10.xx.xx.008 0 0 0 0 0 0 0 0 0 0 0 0
Req Totals: 1 1 0 0 0 2 0 0 0 0 0 0
Req Error %: 0.00 0.00 0.00 0.00 0.00 0.00
------------------------------------------------------------------------------------------------------------------------
Error Summary - All Error Codes
------------------------------------------------------------------------------------------------------------------------
Error Code GET PUT POST HEAD DELETE Total
403 1 1 0 0 0 2
Error Totals 1 1 0 0 0 2
------------------------------------------------------------------------------------------------------------------------
After tracing the 403 errors, the following time skew alert would be seen in logs.
169.254.1.7 dataheadsvc.log.20230413-213831.gz 2023-04-13T15:32:10,334 [qtp1744025389-1494616-0a112bbb:184289e9ad9:fe3c0:4-s3-10.xx.xx.002] ERROR V2Signer.java (line 168) client/server time skewed. RequestId 0a112bbb:184289e9ad9:fe3c0:4Cause
Time is not synchronized with NTP on either Data Domain or ECS.
Resolution
1. Run the latest version of xDoctor to ensure there are no time synchronization or NTP issues reported.
2. Check the system time and hardware clock on all nodes.
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
2. Check the system time and hardware clock on all nodes.
admin@ecsnode1:~> viprexec "hwclock; date"
Output from host : 192.168.219.2
2023-04-14 07:40:03.749250+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.3
2023-04-14 07:40:03.748930+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.7
2023-04-14 07:40:03.747903+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.1
2023-04-14 07:40:03.749030+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.4
2023-04-14 07:40:03.749111+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.8
2023-04-14 07:40:03.748931+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.5
2023-04-14 07:40:03.733340+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.6
2023-04-14 07:40:03.729096+0000
Fri Apr 14 07:40:04 UTC 2023
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
4. If time is synchronized on all ECS nodes, then the issue could be on Data Domain.
5. Engage Data Domain support to check and correct time or NTP issues on Data Domain.
6. Check the cloud unit status.
Affected Products
ECS, Elastic Cloud StorageArticle Properties
Article Number: 000212712
Article Type: Solution
Last Modified: 08 Jun 2023
Version: 2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.