ECS: [atmos_list.py (oc_map)] Object listing

Table of Contents

Detailed Article

Symptoms

Cause

Resolution

Additional Info

Affected Products

Provide Feedback

Summary: List ATMOS objects - "/rest/objects" and "/rest/namespace"

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Check out other resources

Symptoms

Automation through "oc_map"
Inability to perform complete listing with pagination
Lack of support for all listing operations

Cause

Tedious manual efforts
Lack of automation

Resolution

Download "oc_map" (Contains "atmos_list.py" under "~/oc_map/suite/"). Download locations under "Notes" section.
To get usage:
```
#python atmos_list.py -h

	 
```

Examples and Typical Usage:

~/oc_map/suite> python atmos_list.py -h

** ATMOS API usage / example:

   - List objects / OID's:

     --> #python atmos_list.py --atmos_uid <subtenant>/<user> --ss <secret_key>

   - List namespace / object names:

     --> #python atmos_list.py --atmos_uid <subtenant>/<user> --ss <secret_key> --atmos_type namespace --atmos_share <path>

Example output, listing objects

~/oc_map/suite> python atmos_list.py --data_creds atmos_creds.txt

atmos_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:19:07

- ** Option "--quiet" automatically enabled as specific criteria was met.  To disable, restart with option "--quiet_off" **

- Performing ATMOS API call "GET" on file list "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:19:07_get_list" against bucket "626b78f471a44abca3c07d1e2859df46".

  --> Updates will occur every 1000 objects.

  --> Log Location: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:19:07_626b78f471a44abca3c07d1e2859df46_api_req.log".

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:19:07_ob_api_results.log".

    -- Total Objects Scanned: 178
       --> SUCCESS: 0
       --> ERROR:   0
       --> DUR:     0 (Days) 00:00:00.439 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:19:09,  Will now exit.

Example output, listing namespace (shares and folders)

~/oc_map/suite> python atmos_list.py --data_creds atmos_creds.txt --atmos_type namespace --atmos_share /home/admin/oc_map/suite/10-20-2018/ --quiet_off

atmos_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:36:17

- Performing ATMOS API call "GET" on file list "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:36:17_get_list" against bucket "626b78f471a44abca3c07d1e2859df46".

  --> Updates will occur every 1000 objects.

  --> Log Location: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:36:17_626b78f471a44abca3c07d1e2859df46_api_req.log".

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:36:17_ob_api_results.log".

       2f746b50d7f8ad7ab5c23e42012528a8a35f8008cbd8e0e5ba817c99c6bd493f-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_0
       9aa912ddb607636994a9f04f4781ed366d658282aa1f102c3e07381b5728f383-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_1
       2769247420cb789dca33703f6b9167fcfce0dd9e98e829441351b8ed3da254ad-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_10
       5b8bb2177bd2ba12efbf73a42d118247998f690deffd44dd6bda715b12fed452-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_11
       574432e3dfa960927d9bc6453c126f39cf200cc725f0f70b1dae8c7aaf3306ad-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_12
       afbb1d639d575dc3e959222d0c40376a413915911abf72529f3fcbc9d918d1e7-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_13
       cf27160c3213d0fd654ff2ab8cb1f9adcd933b359075c468076784acce8ef0a1-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_14
       f59a9c3a31d037ef58019a2c4819a974e1d6975708910be45f96d30288c44efb-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_15
       ecb8e3ce4615f648d6de4a9277a3e9f59c30c2be8e5a264184da74f38f0d72b3-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_16
       9e0d0f20c3d2a84d41881fee62da7c271e59068893b30247fdefb23ff390cb92-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_17
       88a21a0c6b537928381a0cb3bc789ed6b209fb8699d7c4b0973b92881655d126-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_18
       f187d3b96879586a6cc8d893b915fd6291f35874e176f9bea719b50ea72bd646-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_19
       7a36934594d21db4efebf2035543062277a2eaad8f86a634f8dc6419fcca0b88-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_2
       b88a12cb293a0f599371e02e5bd9bb98c5b00c5b3ce641405c8dfad6cb26ad8c-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_20
       39673763a7919c7c8fdcc9705173fa5b483c93baf04b52c7e3d0349159e2e2ba-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_21
       631623ded51b41dedf936a224220e3b44a27e1eaa96a0e8a0bae7a825b17f2b4-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_22
       22f26c4316a02fa036c45f58dd12f83975eea27a9250910afab5582278af7c26-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_23
       8d2d18175d9a5a6001a7d5d41d86e3095b45bd3ed27dae2ad8efa92ceae3e08b-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_24
       a00cfc00b6441fde4010a88c6f8aa7a2bd5836e96a753c0f38b9925a1ecf60c6-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_25
       aef6f1013dffb14fd98eab730d53f0cc91e1b46b073cec91dc1a367563ecb341-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_26
       9731aa82d710e8cecb75e74058f2c6c4c9edaa5e467252b22527612319d89b28-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_27
       a02c2d0d784e237ff87b2d9eefa620fe07b3f2438bb169da578be1cea00c035f-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_28
       ebdf467e2cdb00d01e5c561a44f970bae0a75d5863438f0d3831d0d372d244cc-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_3
       bd73d4327ee424274339fdb2accae1309caf85fc44adb6687b289fae137b1277-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_4
       4905e59637de695bc48f4f1d629dd9b9c7c3c578701a64f8d1050b11275f863c-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_5
       5a1fe2038f88b130a0f21efa1a4b524f67c174a712d3215d2a578afc03d96583-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_6
       73ff8251870057c1fe1c32075546db9ddb21d570256dd42c9508d247bbb7ef07-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_7
       6f4edd512db356c6f18b74432519b5b43a860dc812701165d43c8fd8e394415b-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_8
       e10d11e8000d0c3744c5af61c9bf28fa0aa3c448d47e4aa3499581152303b483-28e46932-bdf4-4be8-a3ad-7907f1e3bec2 | file_9

    -- Total Objects Scanned: 29
       --> SUCCESS: 0
       --> ERROR:   0
       --> DUR:     0 (Days) 00:00:00.122 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:36:19,  Will now exit.

Example of raw output

~/oc_map/suite> python atmos_list.py --data_creds atmos_creds.txt --atmos_type namespace --atmos_share /home/admin/oc_map/suite/10-20-2018/ --raw

atmos_list_v1.0.0.0 (oc_map_v3.2.3.2), Started: 01-04-2020 17:52:35

- Performing ATMOS API call "GET" on file list "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:52:35_get_list" against bucket "626b78f471a44abca3c07d1e2859df46".

  --> Updates will occur every 1000 objects.

  --> Log Location: "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:52:35_626b78f471a44abca3c07d1e2859df46_api_req.log".

  --> Results Log "/home/admin/oc_map/suite/oc_cache/01-04-2020/17:52:35_ob_api_results.log".

    -- Content:

       <BEGIN>

       <?xml version="1.0" ?>
       <ListDirectoryResponse xmlns="http://www.emc.com/cos/">
         <DirectoryList>
           <DirectoryEntry>
             <ObjectID>2f746b50d7f8ad7ab5c23e42012528a8a35f8008cbd8e0e5ba817c99c6bd493f-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_0</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>9aa912ddb607636994a9f04f4781ed366d658282aa1f102c3e07381b5728f383-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_1</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>2769247420cb789dca33703f6b9167fcfce0dd9e98e829441351b8ed3da254ad-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_10</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>5b8bb2177bd2ba12efbf73a42d118247998f690deffd44dd6bda715b12fed452-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_11</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>574432e3dfa960927d9bc6453c126f39cf200cc725f0f70b1dae8c7aaf3306ad-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_12</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>afbb1d639d575dc3e959222d0c40376a413915911abf72529f3fcbc9d918d1e7-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_13</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>cf27160c3213d0fd654ff2ab8cb1f9adcd933b359075c468076784acce8ef0a1-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_14</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>f59a9c3a31d037ef58019a2c4819a974e1d6975708910be45f96d30288c44efb-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_15</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>ecb8e3ce4615f648d6de4a9277a3e9f59c30c2be8e5a264184da74f38f0d72b3-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_16</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>9e0d0f20c3d2a84d41881fee62da7c271e59068893b30247fdefb23ff390cb92-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_17</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>88a21a0c6b537928381a0cb3bc789ed6b209fb8699d7c4b0973b92881655d126-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_18</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>f187d3b96879586a6cc8d893b915fd6291f35874e176f9bea719b50ea72bd646-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_19</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>7a36934594d21db4efebf2035543062277a2eaad8f86a634f8dc6419fcca0b88-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_2</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>b88a12cb293a0f599371e02e5bd9bb98c5b00c5b3ce641405c8dfad6cb26ad8c-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_20</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>39673763a7919c7c8fdcc9705173fa5b483c93baf04b52c7e3d0349159e2e2ba-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_21</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>631623ded51b41dedf936a224220e3b44a27e1eaa96a0e8a0bae7a825b17f2b4-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_22</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>22f26c4316a02fa036c45f58dd12f83975eea27a9250910afab5582278af7c26-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_23</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>8d2d18175d9a5a6001a7d5d41d86e3095b45bd3ed27dae2ad8efa92ceae3e08b-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_24</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>a00cfc00b6441fde4010a88c6f8aa7a2bd5836e96a753c0f38b9925a1ecf60c6-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_25</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>aef6f1013dffb14fd98eab730d53f0cc91e1b46b073cec91dc1a367563ecb341-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_26</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>9731aa82d710e8cecb75e74058f2c6c4c9edaa5e467252b22527612319d89b28-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_27</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>a02c2d0d784e237ff87b2d9eefa620fe07b3f2438bb169da578be1cea00c035f-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_28</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>ebdf467e2cdb00d01e5c561a44f970bae0a75d5863438f0d3831d0d372d244cc-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_3</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>bd73d4327ee424274339fdb2accae1309caf85fc44adb6687b289fae137b1277-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_4</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>4905e59637de695bc48f4f1d629dd9b9c7c3c578701a64f8d1050b11275f863c-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_5</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>5a1fe2038f88b130a0f21efa1a4b524f67c174a712d3215d2a578afc03d96583-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_6</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>73ff8251870057c1fe1c32075546db9ddb21d570256dd42c9508d247bbb7ef07-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_7</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>6f4edd512db356c6f18b74432519b5b43a860dc812701165d43c8fd8e394415b-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_8</Filename>
           </DirectoryEntry>
           <DirectoryEntry>
             <ObjectID>e10d11e8000d0c3744c5af61c9bf28fa0aa3c448d47e4aa3499581152303b483-28e46932-bdf4-4be8-a3ad-7907f1e3bec2</ObjectID>
             <FileType>regular</FileType>
             <Filename>file_9</Filename>
           </DirectoryEntry>
         </DirectoryList>
       </ListDirectoryResponse>


       <END>


    -- Total Objects Scanned: 29
       --> SUCCESS: 0
       --> ERROR:   0
       --> DUR:     0 (Days) 00:00:00.117 (hh:mm:ss.ms)

- Finished: 01-04-2020 17:52:37,  Will now exit.

Options:

~/oc_map/suite> python atmos_list.py -h

* ATMOS Object Listing:

  --atmos_bkt_dump            List objects
  --ns_mt                     <namespace>, provides ETA during listing
  --max_keys                  <1-1024>, x-emc-limit / pagination
  --chk_versions              Version listing.  Use with options "--atmos_type objects --atmos_share <oid>"
  --api_raw_list              Log with just object name/id
                              --> Used mainly for import in various oc_map tools
                              --> Versioning format: <OID/path>OC::::vId:MAP<version-id>

  -q  --quiet                 Suppress printing
      --quiet_off             Disable automatic quiet
  -w  --w_to_log              Output to default log ** Enabled by default
      --no_log                Disable logging
      --update_cnt            Number between 1 and 1000000000 ("-atmos_bkt_dump" is same as "--max_keys")

      --atmos_uid             <subtenant>/<s3_ecs_user>
      --atmos_ip              <ECS_data_IP>:<9022 or 9023>
      --ss                    s3 secret key

      --atmos_type            <namespace|objects|auto>, Default = auto
                              --> "auto" only works on GET request, and decides when to use "namespace" or "objects"
      --atmos_share           <share name>. Slashes not needed.  Use to create or read from share.  Needed to upload full paths
                              --> Not needed for basename file

      --range_bytes           <bytes-bytes> or <none>, Range in bytes

      --add_header            <x-emc_header:value> add "x-emc" header to new file (avoid spaces)
                              --> Cannot be used with options "--atmos_create_bkt"
                              --> Use with "--api_put" and "--api_put_list"

      --update_header         <x-emc_header:value> add "x-emc" header to existing file or new bucket (avoid spaces)
                              --> Can be used with options "--atmos_create_bkt"
                              --> Use with "--api_put" and "--api_put_list"

      --limited_off           Display Response / Header details on all objects.  Default is to display only on single object

* Advanced / Secure Options:

      --secure_options            Reference KB 517148
      --advanced_options          Advanced options

Additional Information

Latest version of "oc_map" (includes "suite") can be downloaded from:
https://support.emc.com/kb/503854
https://confluence.cec.lab.emc.com/pages/viewpage.action?pageId=35567310
Tool is also available in xDoctor. However, the version may not always be the latest.
WIKI - https://confluence.cec.lab.emc.com/display/ECS/Debugging+commands+using+OC_MAP

This content is translated in 15 languages:

https://downloads.dell.com/TranslatedPDF/CS_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/DA_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/DE_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/ES-XL_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/FI_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/FR_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/IT_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/JA_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/KO_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/NL_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/NO-NO_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/PL_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/PT-BR_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/RU_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/SV_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/TR_KB539999.pdf

https://downloads.dell.com/TranslatedPDF/ZH-CN_KB539999.pdf

Products

ECS Appliance, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption

Article Number: 000051997

Article Type: Solution

Last Modified: 20 Feb 2023

Version: 4

Check if your device is covered by Support Services.

ECS: [atmos_list.py (oc_map)] Object listing

Summary: List ATMOS objects - "/rest/objects" and "/rest/namespace"

Symptoms

Cause

Resolution

Additional Information

Products

Article Properties

Find answers to your questions from other Dell users

Support Services

Article Properties

Find answers to your questions from other Dell users

Support Services