ECS: DD Cloud unit disconnected due to http 403 errors - Access Denied

Zusammenfassung: Cloud unit disconnected due to http 403 errors - Access Denied issue reporting on Data Domain

Dieser Artikel gilt für Dieser Artikel gilt nicht für Dieser Artikel ist nicht an ein bestimmtes Produkt gebunden. In diesem Artikel werden nicht alle Produktversionen aufgeführt.
Andere Ressourcen ansehen

Symptome

Cloud unit gets disconnected due to http 403 errors with Access Denied issue.

The following errors would be reporting on DD. 
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_curl_write_cb:1680 - Request failed with httpcode:403
03/21 07:22:01.838 (tid 0x6991b0): ERROR: CSM cl_request_convert_httpcode_to_err:1554 - HTTP operation returned code:403, request error:Access Denied [5075], uri:https://emcecs.xxxxxxx.xx.xx:9021/?endpoint, date:Tue, 21 Mar 2023 03:22:01 GMT, bytes_sent:0, bytes:rcvd:0
03/21 07:22:01.838 (tid 0x6991b0): ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
03/21 07:22:01 ERROR: csm_provider_version_check:496 - Error (Access Denied) getting the provider version for cloud profile: EMC-ECS
04/03 18:46:49.710 (tid 0x699060): ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied
04/03 18:46:49 ERROR: csm_validate_profile_internal:445 - list_bucket failed for profile cloud-ecs101_profile_profile with err 5075: Access Denied

On ECS, kpi.sh for the affected bucket shows 403 errors. 
Extracting RequestLog data:        DONE

                                 All Requests                                           500 Errors
Node             GETs      PUTs      POSTs     DELETEs   HEADs    Total     GETs    PUTs    POSTs   DELETEs  HEADs   Total
10.xx.xx.001     0         0         0         0         0        0         0       0       0       0        0       0
10.xx.xx.002     0         0         0         0         0        0         0       0       0       0        0       0
10.xx.xx.003     0         0         0         0         0        0         0       0       0       0        0       0
10.xx.xx.004     1         0         0         0         0        1         0       0       0       0        0       0
10.xx.xx.005     0         0         0         0         0        0         0       0       0       0        0       0
10.xx.xx.006     0         0         0         0         0        0         0       0       0       0        0       0
10.xx.xx.007     0         1         0         0         0        1         0       0       0       0        0       0
10.xx.xx.008     0         0         0         0         0        0         0       0       0       0        0       0

Req Totals:      1         1         0         0         0        2         0       0       0       0        0       0
Req Error %:                                                                0.00    0.00    0.00    0.00     0.00    0.00

------------------------------------------------------------------------------------------------------------------------
Error Summary - All Error Codes
------------------------------------------------------------------------------------------------------------------------
Error Code        GET        PUT        POST       HEAD       DELETE     Total

403               1          1          0          0          0          2

Error Totals      1          1          0          0          0          2
------------------------------------------------------------------------------------------------------------------------

After tracing the 403 errors, the following time skew alert would be seen in logs. 
169.254.1.7 dataheadsvc.log.20230413-213831.gz 2023-04-13T15:32:10,334 [qtp1744025389-1494616-0a112bbb:184289e9ad9:fe3c0:4-s3-10.xx.xx.002] ERROR  V2Signer.java (line 168) client/server time skewed. RequestId 0a112bbb:184289e9ad9:fe3c0:4

Ursache

Time is not synchronized with NTP on either Data Domain or ECS.

Lösung

1. Run the latest version of xDoctor to ensure there are no time synchronization or NTP issues reported. 

2. Check the system time and hardware clock on all nodes.  
admin@ecsnode1:~> viprexec "hwclock; date"

Output from host : 192.168.219.2
2023-04-14 07:40:03.749250+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.3
2023-04-14 07:40:03.748930+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.7
2023-04-14 07:40:03.747903+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.1
2023-04-14 07:40:03.749030+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.4
2023-04-14 07:40:03.749111+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.8
2023-04-14 07:40:03.748931+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.5
2023-04-14 07:40:03.733340+0000
Fri Apr 14 07:40:04 UTC 2023

Output from host : 192.168.219.6
2023-04-14 07:40:03.729096+0000
Fri Apr 14 07:40:04 UTC 2023

3. If time sync issue or hardware clock differences are found on ECS nodes, work with Dell Support to resolve them.

4. If time is synchronized on all ECS nodes, then the issue could be on Data Domain.

5. Engage Data Domain support to check and correct time or NTP issues on Data Domain. 

6. Check the cloud unit status.

Betroffene Produkte

ECS, Elastic Cloud Storage
Artikeleigenschaften
Artikelnummer: 000212712
Artikeltyp: Solution
Zuletzt geändert: 08 Juni 2023
Version:  2
Antworten auf Ihre Fragen erhalten Sie von anderen Dell NutzerInnen
Support Services
Prüfen Sie, ob Ihr Gerät durch Support Services abgedeckt ist.