ECS : [s3_list.py (oc_map)] Bucket S3 et liste des versions (liste complète) et vérification ou correction des écarts de suivi d’utilisation
Summary: Bucket S3 et liste des versions (liste complète) et vérifier ou corriger les écarts de suivi d’utilisation.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
- Automatisation via « oc_map »
- À la fin de la liste, l’outil compare la taille totale et le nombre de la liste s3 à ceux du suivi d’utilisation. Un rapport d’intégrité de petite taille est généré
- Incapacité à répertorier le bucket complet (plus de 1 000 objets) par API S3.
- Absence de prise en charge de toutes les opérations de liste.
- Aucune méthode simple pour vérifier les écarts de suivi d’utilisation
Cause
- Efforts manuels fastidieux
- Manque d’automatisation
Resolution
- Téléchargez « oc_map » (contient « s3_list.py » sous « ~/oc_map/suite/ »). Emplacements de téléchargement dans la section « Remarques ».
- Pour obtenir l’utilisation :
#python s3_list.py -h
- Exemples d’utilisation classique :
~/oc_map/suite> python s3_list.py -h ** S3 Bucket List / Dump usage (example): - Run s3 List against one bucket: --> #python s3_list.py --s3_Key --s3_Secret --bucket - Run s3 List against all buckets: --> #python s3_list.py -u root -p ChangeMe --s3_all_bkt --chk_versions --new_map --> ** --quiet can be used on all dumps to suppress output on screen, and only output to log file. - Run s3 MPU List against one bucket: --> #python s3_list.py --s3_Key --s3_Secret --bucket --uploads - To get and fix delta between metering and listing results, add options "--ns_mt " and "mt_fix". Example: --> #python s3_list.py --s3_Key --s3_Secret --bucket --ns_mt --mt_fix - Hybrid listing example (LS Dump, then s3 Prefix (Exact Match) Listing based on each entry from LS dump): --> #python s3_list.py --s3_Key --s3_Secret --hybrid --bypass :
- Exemple de sortie, répertoriant les versions. L’outil détermine quand utiliser « ---chk_versions » pour les buckets compatibles avec la gestion des versions (v3.0.5.2 et versions ultérieures). Peut être désactivé avec « --auto_ver_off » :
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_bkt7 --ns_mt jkns s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:07:16 - ** Option "--quiet" automatically enabled as specific criteria was met. To disable, restart with option "--quiet_off" ** - Preparing round_robin... [RT DT UPDATE 01-04-2020_17:07:18]: Currently 1000 table entries dumped... - Initiating S3 Version listing... - Dumping bucket "jk_bkt7" via S3 API --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp.log". --> Updates will occur every ~1000 objects. --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp_results.log". [UPDATE 01-04-2020_17:07:21]: 1366 of 1389 (98.34% ETA: TBD D, TBD H, Date: TBD)... -- Total Versions: 1366 --> Total Filtered: 1366 --> Current Versions: 262 --> Non-Current Versions: 1104 --> Delete Markers: 395 -- Total Versions Size: 0.04 TB (40.03 GB 40987.10 MB, 41970792.29 KB, 42978091302.00 B) --> Current versions: 0.00 TB (3.01 GB 3080.93 MB, 3154873.30 KB, 3230590260.00 B) --> Non-Current versions: 0.04 TB (37.02 GB 37906.17 MB, 38815918.99 KB, 39747501042.00 B) -- Metering Total Objects: 1389 -- Metering Bucket Size: 0.04 TB (40.06 GB 41019.21 MB, 42003673.00 KB) -- [INFO]: Metering and Bucket size are identical (GB) -- [INFO]: Metering Count > Bucket Count. Deviation within expected range (1000) --> DELTA: -23 Objects --> DUR: 0 (Days) 00:00:01.749 (hh:mm:ss.ms) - Finished: 01-04-2020 17:07:22, Will now exit.
- Exemple de sortie, répertoriant le bucket. L’outil détermine quand utiliser « ---chk_versions » pour les buckets compatibles avec la gestion des versions (v3.0.5.2 et versions ultérieures). Peut être désactivé avec « --auto_ver_off » :
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --quiet_off --screen_off s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:10:39 - Preparing round_robin... - Initiating S3 Bucket listing... - Dumping bucket "jk_no_ver" via S3 API --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp.log". --> Updates will occur every ~1000 objects. --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp_results.log". --> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_all_s3_dmp_results.log". [MTIME]: 2019-12-18T22:09:23.410Z [SIZE]: 4194304 [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_0 [MTIME]: 2019-12-18T22:09:23.498Z [SIZE]: 4194304 [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_1 [MTIME]: 2019-12-04T00:05:56.985Z [SIZE]: 2737 [OBJECT]: test [MTIME]: 2019-12-04T19:46:18.314Z [SIZE]: 7 [OBJECT]: test2 [UPDATE 01-04-2020_17:10:40]: 4 processed... -- Total Objects: 4 -- Total Filtered: 4 -- Total Bucket Size: 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B) -- Metering Total Objects: NA -- Metering Bucket Size: NA --> DUR: 0 (Days) 00:00:00.049 (hh:mm:ss.ms) - Finished: 01-04-2020 17:10:40, Will now exit.
- Exemple de mode de sortie brute :
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --raw
s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:11:53
- Preparing round_robin...
- Initiating S3 Bucket listing...
- Dumping bucket "jk_no_ver" via S3 API
--> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp.log".
--> Updates will occur every ~1000 objects.
--> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp_results.log".
--> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_all_s3_dmp_results.log".
jk_no_ver
1000
false
false
/home/admin/oc_map/suite/oc_cache/12-18-2019/file_0
2019-12-18T22:09:23.410Z
"d0309a6d06e74cf05b10d3b820b9f3d0-4"
4194304
STANDARD
jk_usr
jk_usr
/home/admin/oc_map/suite/oc_cache/12-18-2019/file_1
2019-12-18T22:09:23.498Z
"906c570794b5c7bfd79a4215429002ab-4"
4194304
STANDARD
jk_usr
jk_usr
test
2019-12-04T00:05:56.985Z
"a87d79e111d25c2b5509118c560bd7b0"
2737
STANDARD
jk_usr
jk_usr
test2
2019-12-04T19:46:18.314Z
"e17e69bc2df53a8291233ad25f0ef92d"
7
STANDARD
jk_usr
jk_usr
[UPDATE 01-04-2020_17:11:55]: 4 processed...
-- Total Objects: 4
-- Total Filtered: 4
-- Total Bucket Size: 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B)
-- Metering Total Objects: NA
-- Metering Bucket Size: NA
--> DUR: 0 (Days) 00:00:00.050 (hh:mm:ss.ms)
- Finished: 01-04-2020 17:11:55, Will now exit.
- Options :
~/oc_map/suite> python s3_list.py -h * S3 Bucket List / Dump options: -U --s3_bkt_dump Perform S3 bucket dump (Required ONLY in .oc_map.py) -q --quiet Suppress printing --quiet_off Disable automatic quiet -J --s3_dmp_log File path where output of bucket dump will go. ** Default is "s3_dmp_-
- -_::_.log --> Timestamp is automatically generated for default File Path -w --w_to_log Output to default log ** Enabled by default --no_log Disable logging -K --s3_Key s3 Key / user -L --s3_Secret s3 Secret Key -a --s3_all_bkt Dump all buckets. Requires username (-u --user) to be root or system admin with root rights --> NOTE: In ECS 3.1 buckets owned by root cannot generate s3 secret key. Therefore, these buckets will be skipped -m --bucket bucket -Q --s3_ip End point. Default is local successfully tested Public IP :9020 -W --max_keys Default is 1000 (Recommended). Range 1 - 1000 --> ** NOTE: Anything below 30 can cause inaccuracy of statistics, for later use of "--s3_list_restart" --namespace -A --ns_admin If bucket belongs to namespace admin, this option must be used (--ns_admin ) --chk_versions check all versions (Can also be used on non-versioning buckets) --auto_ver_off Disables automatic detection of s3 versioning. When disabled, "--chk_versions" should be used for version enabled bucket --etag print and/or log etag --ns_mt metering information. Warning will be issued for large deviation --mt_fix Fixes Deltas detected after complete bucket listing. Prompt will appear if detected --> If needed --s3_list_restart can be added to last syntax. Prompt will appear if delta seen in comparison --> Checks for JP Lag first and for exceptions in "metering.log" --> ** Requires EE Approval and Passcode --s3_api_vlist generates list format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> For MPU when using "--uploads" format is "OC::::uId:MAP" --ignore_truncate ignores next marker when performing s3 bucket / version listing --update_cnt , Default = 100000 ("-s3_bkt_dump" is same as "--max_keys") --specific_ns , MUST be used with "--chk_all_bkt". Checks all buckets under specific namespace --mk_bkt_list Makes list of buckets, with credential info (s3Key and secret). List can then be imported with "--import_bkt_list" --> Can also be used with "--specific_ns" switch. --> List can also be manually split up (once complete). Lists can then be used on different nodes. --import_bkt_list , Imports list generated from option "--mk_bkt_list" --dg_vlist generates list of versions preventing cleanup of LS entry on deleted object --> format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> NOTE: This can impact performance of cleanup --dir_chk_1st S3 prefix listing checks if object is a folder --> Without option, object is only determined folder if prefix is found with leading "/" --> Stop cycling on Folder which has child object / version with latest not being delete marker --> Improves performance. Specifically with options "--del_ver_on_del_obj" and "--dg_vlist" --s3_list_eta Rough ETA. Accuracy based on metering --> ** Option "--ns_mt " is required --uploads Initiate MPU (multi-part-upload) listing --upload_id_marker --max_uploads --upload_id --part_number_marker --max_parts --repeat_without_token skips next marker / token on pagination. Repeats until no more token is given *ONLY for troubleshooting* -r --reuse_map Reuse saved map. This will save time by not having to remap. Will generate new map on initial run --> NOTE: This will NOT account for new NS, Buckets, or Users, created after initial map was captured. -n --new_map Create and save map. ** TYPE FILTERS: --cur_only Current versions only --non_cur_only Non-Current versions only --dmarker_only Delete Markers only ** SIZE FILTERS: --size_eq --size_gt --size_lt --size_ge --size_le ** TIME FILTERS: --start , Filter UTC Start time --end , Filter UTC End time ** RECOVERY: --recover_list Generates list of Delete Markers which are latest version --> Can be used with Time Filters "--start" and "--end" --auto_recover Recover automatically by deleting Delete Marker when its latest version * PREFIX, HYBRID and DELIMETER OPTIONS: --delimiter delimiter --prefix list with prefix --hybrid Performs LS Dump, then performs S3 listing using a prefix against each enty in LS dump. --> Also uses unique feature "--exact_match" and automatically ignores dirs --> Great WORKAROUND for Bucket Listing Issues. Multi-threaded for maximum performance --prefix_list List of prefixes (not to be used with --prefix) --exact_match Used with --prefix_list to only return exact match. Meaning noting else starting before or ending after prefix --> Automatically enabled for --hybrid --include_dirs Includes directories / keys ending with "/" --ignore_dirs Used with --prefix_list. Ignores directories / keys ending with "/" --> Automatically enabled for --hybrid ** For "--prefix_list" and "--hybrid" options only: --low_space Only logs entries to "dangling_ls_log" / api logs when there is no corresponding OB or OB analyzer states object is deleted --> Note dump will take longer to complete --ob_exists Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in existing state --ob_deleted Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in deleted state --no_ob Utilizes internal OB Analyzer to filter by LS entries without corresponding OB entry * METADATA SEARCH OPTIONS: --query {selector} {operator} {argument}, for example: "LastModified\\\\\>2015-01-01T00:00:00Z" --attributes is a list one or more metadata names that are not being indexed, but which can be listed in the query results --> For example: --attributes ContentType,Retention --sorted_list , the name of one key that appears in the query expression that becomes the sort key for the query results --include_older_versions , is a boolean that when set to true causes both current and non-current versions of the keys to be listed --> When set to false causes only the current versions of keys to be listed. The default is false. --searchmetadata Returns the list of system and user metadata keys that are currently being indexed for a bucket --> Cannot be used with other search options --raw Raw print. Some tools might append raw to log. Helps if standard output does not print all metadata ** NOTE: Escape characters (5) in "--query" example (\\\\\). These are needed for certain characters such as ">" * Advanced S3 options --s3_list_v1 List version 1. Multiple oc_map s3 tools utilize s3 listing ** Default --s3_list_v2 List version 2. Multiple oc_map s3 tools utilize s3 listing --start_after same as --s3_marker (**s3_list_v2 ONLY) --fetch_owner Fetch owner (**s3_list_v2 ONLY) --s3_sig_v2 Signature 2 --s3_sig_v4 Signature 4 ** Default * RESTART OPTIONS: --s3_marker , object to start listing with --version_id last version ID. Must be used with option "--chk_versions" --s3_list_restart Restarts last failed / stopped listing. Restores statistics and automates all other restart options -u --user user name -p --password password * Advanced / Secure Options: --secure_options Reference KB 517148 --advanced_options Advanced options
Products
ECS Appliance, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Software with Encryption, ECS Appliance Software without EncryptionArticle Properties
Article Number: 000057245
Article Type: Solution
Last Modified: 15 Feb 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.