ECS：[s3_list.py （oc_map）]S3 存储区和版本列表（完整列表）以及检查或修复计量差异
Summary: S3 存储区和版本列表（完整列表）以及检查或修复计量差异。
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
- 通过“oc_map”实现自动化
- 在列表结束时，工具会将 s3 列表中的总大小和计数与计量中的总大小和计数进行比较。 生成小型报告运行状况报告
- 无法按 S3 API 列出完整的存储区（超过 1000 个对象）。
- 缺少对所有列出操作的支持。
- 没有简单的方法来检查计量差异
Cause
- 繁琐的手动工作
- 缺乏自动化
Resolution
- 下载“oc_map”（包含“~/oc_map/suite/”下的“s3_list.py”）。 下载“注释”部分下的位置。
- 要获得用法：
#python s3_list.py -h
- 典型使用情况示例：
~/oc_map/suite> python s3_list.py -h ** S3 Bucket List / Dump usage (example): - Run s3 List against one bucket: --> #python s3_list.py --s3_Key --s3_Secret --bucket - Run s3 List against all buckets: --> #python s3_list.py -u root -p ChangeMe --s3_all_bkt --chk_versions --new_map --> ** --quiet can be used on all dumps to suppress output on screen, and only output to log file. - Run s3 MPU List against one bucket: --> #python s3_list.py --s3_Key --s3_Secret --bucket --uploads - To get and fix delta between metering and listing results, add options "--ns_mt " and "mt_fix". Example: --> #python s3_list.py --s3_Key --s3_Secret --bucket --ns_mt --mt_fix - Hybrid listing example (LS Dump, then s3 Prefix (Exact Match) Listing based on each entry from LS dump): --> #python s3_list.py --s3_Key --s3_Secret --hybrid --bypass :
- 输出示例，列出版本。 工具确定何时将“--chk_versions”用于启用版本控制的存储区（v3.0.5.2 及更高版本）。 可以使用“--auto_ver_off”禁用：
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_bkt7 --ns_mt jkns s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:07:16 - ** Option "--quiet" automatically enabled as specific criteria was met. To disable, restart with option "--quiet_off" ** - Preparing round_robin... [RT DT UPDATE 01-04-2020_17:07:18]: Currently 1000 table entries dumped... - Initiating S3 Version listing... - Dumping bucket "jk_bkt7" via S3 API --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp.log". --> Updates will occur every ~1000 objects. --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:07:16_B1_jk_bkt7_s3_dmp_results.log". [UPDATE 01-04-2020_17:07:21]: 1366 of 1389 (98.34% ETA: TBD D, TBD H, Date: TBD)... -- Total Versions: 1366 --> Total Filtered: 1366 --> Current Versions: 262 --> Non-Current Versions: 1104 --> Delete Markers: 395 -- Total Versions Size: 0.04 TB (40.03 GB 40987.10 MB, 41970792.29 KB, 42978091302.00 B) --> Current versions: 0.00 TB (3.01 GB 3080.93 MB, 3154873.30 KB, 3230590260.00 B) --> Non-Current versions: 0.04 TB (37.02 GB 37906.17 MB, 38815918.99 KB, 39747501042.00 B) -- Metering Total Objects: 1389 -- Metering Bucket Size: 0.04 TB (40.06 GB 41019.21 MB, 42003673.00 KB) -- [INFO]: Metering and Bucket size are identical (GB) -- [INFO]: Metering Count > Bucket Count. Deviation within expected range (1000) --> DELTA: -23 Objects --> DUR: 0 (Days) 00:00:01.749 (hh:mm:ss.ms) - Finished: 01-04-2020 17:07:22, Will now exit.
- 输出示例，列出存储区。工具确定何时将“--chk_versions”用于启用版本控制的存储区（v3.0.5.2 及更高版本）。 可以使用“--auto_ver_off”禁用：
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --quiet_off --screen_off s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:10:39 - Preparing round_robin... - Initiating S3 Bucket listing... - Dumping bucket "jk_no_ver" via S3 API --> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp.log". --> Updates will occur every ~1000 objects. --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_B1_jk_no_ver_s3_dmp_results.log". --> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:10:38_all_s3_dmp_results.log". [MTIME]: 2019-12-18T22:09:23.410Z [SIZE]: 4194304 [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_0 [MTIME]: 2019-12-18T22:09:23.498Z [SIZE]: 4194304 [OBJECT]: /home/admin/oc_map/suite/oc_cache/12-18-2019/file_1 [MTIME]: 2019-12-04T00:05:56.985Z [SIZE]: 2737 [OBJECT]: test [MTIME]: 2019-12-04T19:46:18.314Z [SIZE]: 7 [OBJECT]: test2 [UPDATE 01-04-2020_17:10:40]: 4 processed... -- Total Objects: 4 -- Total Filtered: 4 -- Total Bucket Size: 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B) -- Metering Total Objects: NA -- Metering Bucket Size: NA --> DUR: 0 (Days) 00:00:00.049 (hh:mm:ss.ms) - Finished: 01-04-2020 17:10:40, Will now exit.
- 原始输出模式示例：
~/oc_map/suite> python s3_list.py --data_creds s3_creds.txt --bucket jk_no_ver --raw
s3_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:11:53
- Preparing round_robin...
- Initiating S3 Bucket listing...
- Dumping bucket "jk_no_ver" via S3 API
--> Log File: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp.log".
--> Updates will occur every ~1000 objects.
--> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_B1_jk_no_ver_s3_dmp_results.log".
--> Results Log for all buckets "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:11:53_all_s3_dmp_results.log".
jk_no_ver
1000
false
false
/home/admin/oc_map/suite/oc_cache/12-18-2019/file_0
2019-12-18T22:09:23.410Z
"d0309a6d06e74cf05b10d3b820b9f3d0-4"
4194304
STANDARD
jk_usr
jk_usr
/home/admin/oc_map/suite/oc_cache/12-18-2019/file_1
2019-12-18T22:09:23.498Z
"906c570794b5c7bfd79a4215429002ab-4"
4194304
STANDARD
jk_usr
jk_usr
test
2019-12-04T00:05:56.985Z
"a87d79e111d25c2b5509118c560bd7b0"
2737
STANDARD
jk_usr
jk_usr
test2
2019-12-04T19:46:18.314Z
"e17e69bc2df53a8291233ad25f0ef92d"
7
STANDARD
jk_usr
jk_usr
[UPDATE 01-04-2020_17:11:55]: 4 processed...
-- Total Objects: 4
-- Total Filtered: 4
-- Total Bucket Size: 0.00 TB (0.01 GB 8.00 MB, 8194.68 KB, 8391352.00 B)
-- Metering Total Objects: NA
-- Metering Bucket Size: NA
--> DUR: 0 (Days) 00:00:00.050 (hh:mm:ss.ms)
- Finished: 01-04-2020 17:11:55, Will now exit.
- 选项包括：
~/oc_map/suite> python s3_list.py -h * S3 Bucket List / Dump options: -U --s3_bkt_dump Perform S3 bucket dump (Required ONLY in .oc_map.py) -q --quiet Suppress printing --quiet_off Disable automatic quiet -J --s3_dmp_log File path where output of bucket dump will go. ** Default is "s3_dmp_-
- -_::_.log --> Timestamp is automatically generated for default File Path -w --w_to_log Output to default log ** Enabled by default --no_log Disable logging -K --s3_Key s3 Key / user -L --s3_Secret s3 Secret Key -a --s3_all_bkt Dump all buckets. Requires username (-u --user) to be root or system admin with root rights --> NOTE: In ECS 3.1 buckets owned by root cannot generate s3 secret key. Therefore, these buckets will be skipped -m --bucket bucket -Q --s3_ip End point. Default is local successfully tested Public IP :9020 -W --max_keys Default is 1000 (Recommended). Range 1 - 1000 --> ** NOTE: Anything below 30 can cause inaccuracy of statistics, for later use of "--s3_list_restart" --namespace -A --ns_admin If bucket belongs to namespace admin, this option must be used (--ns_admin ) --chk_versions check all versions (Can also be used on non-versioning buckets) --auto_ver_off Disables automatic detection of s3 versioning. When disabled, "--chk_versions" should be used for version enabled bucket --etag print and/or log etag --ns_mt metering information. Warning will be issued for large deviation --mt_fix Fixes Deltas detected after complete bucket listing. Prompt will appear if detected --> If needed --s3_list_restart can be added to last syntax. Prompt will appear if delta seen in comparison --> Checks for JP Lag first and for exceptions in "metering.log" --> ** Requires EE Approval and Passcode --s3_api_vlist generates list format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> For MPU when using "--uploads" format is "OC::::uId:MAP" --ignore_truncate ignores next marker when performing s3 bucket / version listing --update_cnt , Default = 100000 ("-s3_bkt_dump" is same as "--max_keys") --specific_ns , MUST be used with "--chk_all_bkt". Checks all buckets under specific namespace --mk_bkt_list Makes list of buckets, with credential info (s3Key and secret). List can then be imported with "--import_bkt_list" --> Can also be used with "--specific_ns" switch. --> List can also be manually split up (once complete). Lists can then be used on different nodes. --import_bkt_list , Imports list generated from option "--mk_bkt_list" --dg_vlist generates list of versions preventing cleanup of LS entry on deleted object --> format "OC::::vId:MAP" to be used with S3 API list calls "GET/HEAD/DELETE" --> NOTE: This can impact performance of cleanup --dir_chk_1st S3 prefix listing checks if object is a folder --> Without option, object is only determined folder if prefix is found with leading "/" --> Stop cycling on Folder which has child object / version with latest not being delete marker --> Improves performance. Specifically with options "--del_ver_on_del_obj" and "--dg_vlist" --s3_list_eta Rough ETA. Accuracy based on metering --> ** Option "--ns_mt " is required --uploads Initiate MPU (multi-part-upload) listing --upload_id_marker --max_uploads --upload_id --part_number_marker --max_parts --repeat_without_token skips next marker / token on pagination. Repeats until no more token is given *ONLY for troubleshooting* -r --reuse_map Reuse saved map. This will save time by not having to remap. Will generate new map on initial run --> NOTE: This will NOT account for new NS, Buckets, or Users, created after initial map was captured. -n --new_map Create and save map. ** TYPE FILTERS: --cur_only Current versions only --non_cur_only Non-Current versions only --dmarker_only Delete Markers only ** SIZE FILTERS: --size_eq --size_gt --size_lt --size_ge --size_le ** TIME FILTERS: --start , Filter UTC Start time --end , Filter UTC End time ** RECOVERY: --recover_list Generates list of Delete Markers which are latest version --> Can be used with Time Filters "--start" and "--end" --auto_recover Recover automatically by deleting Delete Marker when its latest version * PREFIX, HYBRID and DELIMETER OPTIONS: --delimiter delimiter --prefix list with prefix --hybrid Performs LS Dump, then performs S3 listing using a prefix against each enty in LS dump. --> Also uses unique feature "--exact_match" and automatically ignores dirs --> Great WORKAROUND for Bucket Listing Issues. Multi-threaded for maximum performance --prefix_list List of prefixes (not to be used with --prefix) --exact_match Used with --prefix_list to only return exact match. Meaning noting else starting before or ending after prefix --> Automatically enabled for --hybrid --include_dirs Includes directories / keys ending with "/" --ignore_dirs Used with --prefix_list. Ignores directories / keys ending with "/" --> Automatically enabled for --hybrid ** For "--prefix_list" and "--hybrid" options only: --low_space Only logs entries to "dangling_ls_log" / api logs when there is no corresponding OB or OB analyzer states object is deleted --> Note dump will take longer to complete --ob_exists Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in existing state --ob_deleted Utilizes internal OB Analyzer to filter by LS entries with corresponding OB entry in deleted state --no_ob Utilizes internal OB Analyzer to filter by LS entries without corresponding OB entry * METADATA SEARCH OPTIONS: --query {selector} {operator} {argument}, for example: "LastModified\\\\\>2015-01-01T00:00:00Z" --attributes is a list one or more metadata names that are not being indexed, but which can be listed in the query results --> For example: --attributes ContentType,Retention --sorted_list , the name of one key that appears in the query expression that becomes the sort key for the query results --include_older_versions , is a boolean that when set to true causes both current and non-current versions of the keys to be listed --> When set to false causes only the current versions of keys to be listed. The default is false. --searchmetadata Returns the list of system and user metadata keys that are currently being indexed for a bucket --> Cannot be used with other search options --raw Raw print. Some tools might append raw to log. Helps if standard output does not print all metadata ** NOTE: Escape characters (5) in "--query" example (\\\\\). These are needed for certain characters such as ">" * Advanced S3 options --s3_list_v1 List version 1. Multiple oc_map s3 tools utilize s3 listing ** Default --s3_list_v2 List version 2. Multiple oc_map s3 tools utilize s3 listing --start_after same as --s3_marker (**s3_list_v2 ONLY) --fetch_owner Fetch owner (**s3_list_v2 ONLY) --s3_sig_v2 Signature 2 --s3_sig_v4 Signature 4 ** Default * RESTART OPTIONS: --s3_marker , object to start listing with --version_id last version ID. Must be used with option "--chk_versions" --s3_list_restart Restarts last failed / stopped listing. Restores statistics and automates all other restart options -u --user user name -p --password password * Advanced / Secure Options: --secure_options Reference KB 517148 --advanced_options Advanced options
Products
ECS Appliance, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Software with Encryption, ECS Appliance Software without EncryptionArticle Properties
Article Number: 000057245
Article Type: Solution
Last Modified: 15 Feb 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.