ECS: [s3_list.py (oc_map)] Сегмент S3 і список версій (повний список) і Перевірка або виправлення розбіжностей у вимірюванні

Автоматизація через "oc_map"
Наприкінці списку інструмент порівнює загальний розмір і кількість зі списку s3 зі списку з показниками з вимірювання. Формується невеликий звіт про стан здоров'я

Неможливість перерахувати повний сегмент (більше 1000 об'єктів) за допомогою S3 API.
Відсутність підтримки всіх операцій лістингу.
Немає простого методу перевірки розбіжностей у вимірюванні

Виснажливі ручні зусилля
відсутність автоматизації

Завантажте "oc_map" (Містить "s3_list.py" у розділі "~/oc_map/suite/"). Завантажте локації в розділі "Примітки".
Щоб отримати використання:
```
#python s3_list.py -h

	
```

Приклади типового використання:

~/oc_map/suite> python s3_list.py -h

** S3 Bucket List / Dump usage (example):

   - Run s3 List against one bucket:

     --> #python s3_list.py --s3_Key  --s3_Secret  --bucket 

   - Run s3 List against all buckets:

     --> #python s3_list.py -u root -p ChangeMe --s3_all_bkt --chk_versions --new_map

         --> ** --quiet can be used on all dumps to suppress output on screen, and only output to log file.

   - Run s3 MPU List against one bucket:

     --> #python s3_list.py --s3_Key  --s3_Secret  --bucket  --uploads

   - To get and fix delta between metering and listing results, add options "--ns_mt " and "mt_fix".  Example:

         --> #python s3_list.py --s3_Key  --s3_Secret  --bucket  --ns_mt  --mt_fix

   - Hybrid listing example (LS Dump, then s3 Prefix (Exact Match) Listing based on each entry from LS dump):

         --> #python s3_list.py --s3_Key  --s3_Secret  --hybrid --bypass :

Приклад виводу, перерахування версій. Засіб визначає, коли слід використовувати "--chk_versions" для сегментів із підтримкою керування версіями (v3.0.5.2 і новіших версій). Можна відключити за допомогою "--auto_ver_off":

~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_bkt7 --ns_mt jkns

s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:07:16

- ** Option "--quiet" automatically enabled as specific criteria was met.  To disable, restart with option "--quiet_off" **

- Preparing round_robin...

    [RT DT UPDATE 01-04-2020_17:07:18]: Currently 1000 table entries dumped...

- Initiating S3 Version listing...

- Dumping bucket "jk_bkt7" via S3 API

  --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp.log".

  --> Updates will occur every ~1000 objects.

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp_results.log".

    [UPDATE 01-04-2020_17:07:21]: 1366 of 1389 (98.34% ETA: TBD D, TBD H, Date: TBD)...

    -- Total Versions:                    1366

       --> Total Filtered:                1366

       --> Current Versions:              262

       --> Non-Current Versions:          1104

       --> Delete Markers:                395

    -- Total Versions Size:               0.04 TB (40.03 GB 40987.10 MB, 41970792.29 KB, 42978091302.00 B)

       --> Current versions:              0.00 TB (3.01 GB 3080.93 MB, 3154873.30 KB, 3230590260.00 B)

       --> Non-Current versions:          0.04 TB (37.02 GB 37906.17 MB, 38815918.99 KB, 39747501042.00 B)

    -- Metering Total Objects:            1389

    -- Metering Bucket Size:              0.04 TB (40.06 GB 41019.21 MB, 42003673.00 KB)

    -- [INFO]: Metering and Bucket size are identical (GB)

    -- [INFO]: Metering Count > Bucket Count.  Deviation within expected range (1000)

       --> DELTA: -23 Objects

    --> DUR:                              0 (Days) 00:00:01.749 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:07:22,  Will now exit.

Приклад виводу, сегмент списку. Засіб визначає, коли слід використовувати "--chk_versions" для сегментів із підтримкою керування версіями (v3.0.5.2 і новіших версій). Можна відключити за допомогою "--auto_ver_off":

~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --quiet_off --screen_off

s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:10:39

- Preparing round_robin...

- Initiating S3 Bucket listing...

- Dumping bucket "jk_no_ver" via S3 API

  --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp.log".

  --> Updates will occur every ~1000 objects.

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp_results.log".

  --> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_all_s3_dmp_results.log".

    [MTIME]: 2019-12-18T22:09:23.410Z [SIZE]: 4194304          [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_0
    [MTIME]: 2019-12-18T22:09:23.498Z [SIZE]: 4194304          [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_1
    [MTIME]: 2019-12-04T00:05:56.985Z [SIZE]: 2737             [OBJECT]: test
    [MTIME]: 2019-12-04T19:46:18.314Z [SIZE]: 7                [OBJECT]: test2

    [UPDATE 01-04-2020_17:10:40]: 4 processed...

    -- Total Objects:                     4

    -- Total Filtered:                    4

    -- Total Bucket Size:                 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B)

    -- Metering Total Objects:            NA

    -- Metering Bucket Size:              NA

    --> DUR:                              0 (Days) 00:00:00.049 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:10:40,  Will now exit.

Приклад режиму необробленого виводу:

~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --raw

s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:11:53

- Preparing round_robin...

- Initiating S3 Bucket listing...

- Dumping bucket "jk_no_ver" via S3 API

  --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp.log".

  --> Updates will occur every ~1000 objects.

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp_results.log".

  --> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_all_s3_dmp_results.log".



  jk_no_ver
  
  
  1000
  false
  false
  
    /home/admin/oc_map/suite/oc_cache/12-18-2019/file_0
    2019-12-18T22:09:23.410Z
    "d0309a6d06e74cf05b10d3b820b9f3d0-4"
    4194304
    STANDARD
    
      jk_usr
      jk_usr
    
  
  
    /home/admin/oc_map/suite/oc_cache/12-18-2019/file_1
    2019-12-18T22:09:23.498Z
    "906c570794b5c7bfd79a4215429002ab-4"
    4194304
    STANDARD
    
      jk_usr
      jk_usr
    
  
  
    test
    2019-12-04T00:05:56.985Z
    "a87d79e111d25c2b5509118c560bd7b0"
    2737
    STANDARD
    
      jk_usr
      jk_usr
    
  
  
    test2
    2019-12-04T19:46:18.314Z
    "e17e69bc2df53a8291233ad25f0ef92d"
    7
    STANDARD
    
      jk_usr
      jk_usr
    
  



    [UPDATE 01-04-2020_17:11:55]: 4 processed...

    -- Total Objects:                     4

    -- Total Filtered:                    4

    -- Total Bucket Size:                 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B)

    -- Metering Total Objects:            NA

    -- Metering Bucket Size:              NA

    --> DUR:                              0 (Days) 00:00:00.050 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:11:55,  Will now exit.

Параметри:
```
~/oc_map/suite> python s3_list.py -h

* S3 Bucket List / Dump options:

  -U  --s3_bkt_dump               Perform S3 bucket dump (Required ONLY in .oc_map.py)
  -q  --quiet                     Suppress printing
      --quiet_off                 Disable automatic quiet
  -J  --s3_dmp_log                File path where output of bucket dump will go.  ** Default is "s3_dmp_-
```
-_::_.log --> Timestamp is automatically generated for default File Path -w --w_to_log Output to default log ** Enabled by default --no_log Disable logging -K --s3_Key s3 Key / user -L --s3_Secret s3 Secret Key -a --s3_all_bkt Dump all buckets. Requires username (-u --user) to be root or system admin with root rights --> NOTE: In ECS 3.1 buckets owned by root cannot generate s3 secret key. Therefore, these buckets will be skipped -m --bucket bucket -Q --s3_ip End point. Default is local successfully tested Public IP :9020 -W --max_keys Default is 1000 (Recommended). Range 1 - 1000 --> ** NOTE: Anything below 30 can cause inaccuracy of statistics, for later use of "--s3_list_restart" --namespace -A --ns_admin If bucket belongs to namespace admin, this option must be used (--ns_admin ) --chk_versions check all versions (Can also be used on non-versioning buckets) --auto_ver_off Disables automatic detection of s3 versioning. When disabled, "--chk_versions" should be used for version enabled bucket --etag print and/or log etag --ns_mt metering information. Warning will be issued for large deviation --mt_fix Fixes Deltas detected after complete bucket listing. Prompt will appear if detected --> If needed --s3_list_restart can be added to last syntax. Prompt will appear if delta seen in comparison --> Checks for JP Lag first and for exceptions in "metering.log" --> ** Requires EE Approval and Passcode --s3_api_vlist generates list format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> For MPU when using "--uploads" format is "OC::::uId:MAP" --ignore_truncate ignores next marker when performing s3 bucket / version listing --update_cnt , Default = 100000 ("-s3_bkt_dump" is same as "--max_keys") --specific_ns , MUST be used with "--chk_all_bkt". Checks all buckets under specific namespace --mk_bkt_list Makes list of buckets, with credential info (s3Key and secret). List can then be imported with "--import_bkt_list" --> Can also be used with "--specific_ns" switch. --> List can also be manually split up (once complete). Lists can then be used on different nodes. --import_bkt_list , Imports list generated from option "--mk_bkt_list" --dg_vlist generates list of versions preventing cleanup of LS entry on deleted object --> format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> NOTE: This can impact performance of cleanup --dir_chk_1st S3 prefix listing checks if object is a folder --> Without option, object is only determined folder if prefix is found with leading "/" --> Stop cycling on Folder which has child object / version with latest not being delete marker --> Improves performance. Specifically with options "--del_ver_on_del_obj" and "--dg_vlist" --s3_list_eta Rough ETA. Accuracy based on metering --> ** Option "--ns_mt " is required --uploads Initiate MPU (multi-part-upload) listing --upload_id_marker --max_uploads --upload_id --part_number_marker --max_parts --repeat_without_token skips next marker / token on pagination. Repeats until no more token is given *ONLY for troubleshooting* -r --reuse_map Reuse saved map. This will save time by not having to remap. Will generate new map on initial run --> NOTE: This will NOT account for new NS, Buckets, or Users, created after initial map was captured. -n --new_map Create and save map. ** TYPE FILTERS: --cur_only Current versions only --non_cur_only Non-Current versions only --dmarker_only Delete Markers only ** SIZE FILTERS: --size_eq --size_gt --size_lt --size_ge --size_le ** TIME FILTERS: --start , Filter UTC Start time --end , Filter UTC End time ** RECOVERY: --recover_list Generates list of Delete Markers which are latest version --> Can be used with Time Filters "--start" and "--end" --auto_recover Recover automatically by deleting Delete Marker when its latest version * PREFIX, HYBRID and DELIMETER OPTIONS: --delimiter delimiter --prefix list with prefix --hybrid Performs LS Dump, then performs S3 listing using a prefix against each enty in LS dump. --> Also uses unique feature "--exact_match" and automatically ignores dirs --> Great WORKAROUND for Bucket Listing Issues. Multi-threaded for maximum performance --prefix_list List of prefixes (not to be used with --prefix) --exact_match Used with --prefix_list to only return exact match. Meaning noting else starting before or ending after prefix --> Automatically enabled for --hybrid --include_dirs Includes directories / keys ending with "/" --ignore_dirs Used with --prefix_list. Ignores directories / keys ending with "/" --> Automatically enabled for --hybrid ** For "--prefix_list" and "--hybrid" options only: --low_space Only logs entries to "dangling_ls_log" / api logs when there is no corresponding OB or OB analyzer states object is deleted --> Note dump will take longer to complete --ob_exists Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in existing state --ob_deleted Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in deleted state --no_ob Utilizes internal OB Analyzer to filter by LS entries without corresponding OB entry * METADATA SEARCH OPTIONS: --query {selector} {operator} {argument}, for example: "LastModified\\\\\>2015-01-01T00:00:00Z" --attributes is a list one or more metadata names that are not being indexed, but which can be listed in the query results --> For example: --attributes ContentType,Retention --sorted_list , the name of one key that appears in the query expression that becomes the sort key for the query results --include_older_versions , is a boolean that when set to true causes both current and non-current versions of the keys to be listed --> When set to false causes only the current versions of keys to be listed. The default is false. --searchmetadata Returns the list of system and user metadata keys that are currently being indexed for a bucket --> Cannot be used with other search options --raw Raw print. Some tools might append raw to log. Helps if standard output does not print all metadata ** NOTE: Escape characters (5) in "--query" example (\\\\\). These are needed for certain characters such as ">" * Advanced S3 options --s3_list_v1 List version 1. Multiple oc_map s3 tools utilize s3 listing ** Default --s3_list_v2 List version 2. Multiple oc_map s3 tools utilize s3 listing --start_after same as --s3_marker (**s3_list_v2 ONLY) --fetch_owner Fetch owner (**s3_list_v2 ONLY) --s3_sig_v2 Signature 2 --s3_sig_v4 Signature 4 ** Default * RESTART OPTIONS: --s3_marker , object to start listing with --version_id last version ID. Must be used with option "--chk_versions" --s3_list_restart Restarts last failed / stopped listing. Restores statistics and automates all other restart options -u --user user name -p --password password * Advanced / Secure Options: --secure_options Reference KB 517148 --advanced_options Advanced options

ECS: [s3_list.py (oc_map)] Сегмент S3 і список версій (повний список) і Перевірка або виправлення розбіжностей у вимірюванні

Summary: Сегмент S3 і список версій (повний список) і Перевірка або виправлення розбіжностей у вимірюванні.

Symptoms

Cause

Resolution

Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services

ECS: [s3_list.py (oc_map)] Сегмент S3 і список версій (повний список) і Перевірка або виправлення розбіжностей у вимірюванні

Summary: Сегмент S3 і список версій (повний список) і Перевірка або виправлення розбіжностей у вимірюванні.

Detailed Article

Symptoms

Cause

Resolution

Affected Products

Symptoms

Cause

Resolution

Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services