ECS: DD Cloud unit disconnected due to http 403 errors - Access Denied
摘要: Cloud unit disconnected due to http 403 errors - Access Denied issue reporting on Data Domain
本文章適用於
本文章不適用於
本文無關於任何特定產品。
本文未識別所有產品版本。
症狀
Cloud unit gets disconnected due to http 403 errors with Access Denied issue.
The following errors would be reporting on DD.
On ECS, kpi.sh for the affected bucket shows 403 errors.
After tracing the 403 errors, the following time skew alert would be seen in logs.
The following errors would be reporting on DD.
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_curl_write_cb:1680 - Request failed with httpcode:403
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_request_convert_httpcode_to_err:1554 - HTTP operation returned code:403, request error:Access Denied [5075], uri:https://emcecs.xxxxxxx.xx.xx:9021/?endpoint, date:Tue, 21 Mar 2023 03:22:01 GMT, bytes_sent:0, bytes:rcvd:0
03/21 07:22:01.838 (tid 0x6991b0): ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
03/21 07:22:01 ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
04/03 18:46:49.710 (tid 0x699060): ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
04/03 18:46:49 ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
On ECS, kpi.sh for the affected bucket shows 403 errors.
Extracting RequestLog data: DONE
All Requests 500 Errors
Node GETs PUTs POSTs DELETEs HEADs Total GETs PUTs POSTs DELETEs HEADs Total
10.xx.xx.001 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.002 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.003 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.004 1 0 0 0 0 1 0 0 0 0 0 0
10.xx.xx.005 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.006 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.007 0 1 0 0 0 1 0 0 0 0 0 0
10.xx.xx.008 0 0 0 0 0 0 0 0 0 0 0 0
Req Totals: 1 1 0 0 0 2 0 0 0 0 0 0
Req Error %: 0.00 0.00 0.00 0.00 0.00 0.00
------------------------------------------------------------------------------------------------------------------------
Error Summary - All Error Codes
------------------------------------------------------------------------------------------------------------------------
Error Code GET PUT POST HEAD DELETE Total
403 1 1 0 0 0 2
Error Totals 1 1 0 0 0 2
------------------------------------------------------------------------------------------------------------------------
After tracing the 403 errors, the following time skew alert would be seen in logs.
169.254.1.7 dataheadsvc.log.20230413-213831.gz 2023-04-13T15:32:10,334 [qtp1744025389-1494616-0a112bbb:184289e9ad9:fe3c0:4-s3-10.xx.xx.002] ERROR V2Signer.java (line 168) client/server time skewed. RequestId 0a112bbb:184289e9ad9:fe3c0:4原因
Time is not synchronized with NTP on either Data Domain or ECS.
解析度
1. Run the latest version of xDoctor to ensure there are no time synchronization or NTP issues reported.
2. Check the system time and hardware clock on all nodes.
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
2. Check the system time and hardware clock on all nodes.
admin@ecsnode1:~> viprexec "hwclock; date"
Output from host : 192.168.219.2
2023-04-14 07:40:03.749250+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.3
2023-04-14 07:40:03.748930+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.7
2023-04-14 07:40:03.747903+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.1
2023-04-14 07:40:03.749030+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.4
2023-04-14 07:40:03.749111+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.8
2023-04-14 07:40:03.748931+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.5
2023-04-14 07:40:03.733340+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.6
2023-04-14 07:40:03.729096+0000
Fri Apr 14 07:40:04 UTC 2023
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
4. If time is synchronized on all ECS nodes, then the issue could be on Data Domain.
5. Engage Data Domain support to check and correct time or NTP issues on Data Domain.
6. Check the cloud unit status.
受影響的產品
ECS, Elastic Cloud Storage文章屬性
文章編號: 000212712
文章類型: Solution
上次修改時間: 08 6月 2023
版本: 2
向其他 Dell 使用者尋求您問題的答案
支援服務
檢查您的裝置是否在支援服務的涵蓋範圍內。