ECS: DD Cloud unit disconnected due to http 403 errors - Access Denied
概要: Cloud unit disconnected due to http 403 errors - Access Denied issue reporting on Data Domain
この記事は次に適用されます:
この記事は次には適用されません:
この記事は、特定の製品に関連付けられていません。
すべての製品パージョンがこの記事に記載されているわけではありません。
現象
Cloud unit gets disconnected due to http 403 errors with Access Denied issue.
The following errors would be reporting on DD.
On ECS, kpi.sh for the affected bucket shows 403 errors.
After tracing the 403 errors, the following time skew alert would be seen in logs.
The following errors would be reporting on DD.
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_curl_write_cb:1680 - Request failed with httpcode:403
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_request_convert_httpcode_to_err:1554 - HTTP operation returned code:403, request error:Access Denied [5075], uri:https://emcecs.xxxxxxx.xx.xx:9021/?endpoint, date:Tue, 21 Mar 2023 03:22:01 GMT, bytes_sent:0, bytes:rcvd:0
03/21 07:22:01.838 (tid 0x6991b0): ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
03/21 07:22:01 ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
04/03 18:46:49.710 (tid 0x699060): ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
04/03 18:46:49 ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
On ECS, kpi.sh for the affected bucket shows 403 errors.
Extracting RequestLog data: DONE
All Requests 500 Errors
Node GETs PUTs POSTs DELETEs HEADs Total GETs PUTs POSTs DELETEs HEADs Total
10.xx.xx.001 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.002 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.003 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.004 1 0 0 0 0 1 0 0 0 0 0 0
10.xx.xx.005 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.006 0 0 0 0 0 0 0 0 0 0 0 0
10.xx.xx.007 0 1 0 0 0 1 0 0 0 0 0 0
10.xx.xx.008 0 0 0 0 0 0 0 0 0 0 0 0
Req Totals: 1 1 0 0 0 2 0 0 0 0 0 0
Req Error %: 0.00 0.00 0.00 0.00 0.00 0.00
------------------------------------------------------------------------------------------------------------------------
Error Summary - All Error Codes
------------------------------------------------------------------------------------------------------------------------
Error Code GET PUT POST HEAD DELETE Total
403 1 1 0 0 0 2
Error Totals 1 1 0 0 0 2
------------------------------------------------------------------------------------------------------------------------
After tracing the 403 errors, the following time skew alert would be seen in logs.
169.254.1.7 dataheadsvc.log.20230413-213831.gz 2023-04-13T15:32:10,334 [qtp1744025389-1494616-0a112bbb:184289e9ad9:fe3c0:4-s3-10.xx.xx.002] ERROR V2Signer.java (line 168) client/server time skewed. RequestId 0a112bbb:184289e9ad9:fe3c0:4原因
Time is not synchronized with NTP on either Data Domain or ECS.
解決方法
1. Run the latest version of xDoctor to ensure there are no time synchronization or NTP issues reported.
2. Check the system time and hardware clock on all nodes.
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
2. Check the system time and hardware clock on all nodes.
admin@ecsnode1:~> viprexec "hwclock; date"
Output from host : 192.168.219.2
2023-04-14 07:40:03.749250+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.3
2023-04-14 07:40:03.748930+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.7
2023-04-14 07:40:03.747903+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.1
2023-04-14 07:40:03.749030+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.4
2023-04-14 07:40:03.749111+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.8
2023-04-14 07:40:03.748931+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.5
2023-04-14 07:40:03.733340+0000
Fri Apr 14 07:40:04 UTC 2023
Output from host : 192.168.219.6
2023-04-14 07:40:03.729096+0000
Fri Apr 14 07:40:04 UTC 2023
3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.
4. If time is synchronized on all ECS nodes, then the issue could be on Data Domain.
5. Engage Data Domain support to check and correct time or NTP issues on Data Domain.
6. Check the cloud unit status.
対象製品
ECS, Elastic Cloud Storage文書のプロパティ
文書番号: 000212712
文書の種類: Solution
最終更新: 08 6月 2023
バージョン: 2
質問に対する他のDellユーザーからの回答を見つける
サポート サービス
お使いのデバイスがサポート サービスの対象かどうかを確認してください。