UNSOLVED

danielsj1

updated

6 years ago

D

danielsj1

3 Posts

0

5839

October 22nd, 2018 08:00

Delete large bucket

If I have a bucket with a very large number of files, what is the best way to recursively delete the objects so I can delete the bucket?

With s3cmd, it's so large recursive delete fails. For one of the smaller buckets I'm doing s3cmd ls + s3cmd del using gnu parallel. But the larger bucket is 100,000,000+ objects and this method doesn't work.

  • danielsj1

    3 Posts

    5098

    0

    Posted October 23rd, 2018 12:00

    I can’t view that page as it says dell internal only.

    “

    If you are an External user and require access to inside Dell then please contact your Dell representative for more information. Please note External User access is limited to Dell approved customers, partners, and suppliers.

    “

    --

    John Daniels

    Research IS Systems Engineer III

    Children's Hospital of Philadelphia Research Institute

    The Roberts Center for Pediatric Research

    2716 South Street

    Philadelphia, PA 19104

    danielsj1@email.chop.edu

    Office: 267-425-0155

    Need faster service?

    Try placing your request at http://cirrus.research.chop.edu before opening a manual service request.

    Otherwise, you can submit an adhoc request here (general->other), include ‘please route to research IS’

  • chris_arnett

    1 Rookie

    •

    110 Posts

    5098

    0

    Posted October 25th, 2018 13:00

    You can get the bucket-wipe tool here:

    WARNING: This will erase the bucket and all of its data!  Please make absolutely sure this is what you want.

    http://130753149435015067.public.ecstestdrive.com/share/bucket-wipe-1.9.jar


    usage: java -jar bucket-wipe.jar [options]

    -a,--access-key                    the S3 access key

    -e,--endpoint                      the endpoint to connect to, including

                                        protocol, host, and port

    -h,--help                          displays this help text

    -hier,--hierarchical               Enumerate the bucket hierarchically.  This

                                        is recommended for ECS's

                                        filesystem-enabled buckets.

        --keep-bucket                   do not delete the bucket when done

    -l,--key-list                      instead of listing bucket, delete objects

                                        matched in source file key list

        --no-smart-client               disables the ECS smart-client. use this

                                        option with an external load balancer

    -p,--prefix                        deletes only objects under the specified

                                        prefix

    -s,--secret-key                    the secret key

        --stacktrace                    displays full stack trace of errors

    -t,--threads                       number of threads to use

        --vhost                         enables DNS buckets and turns off load

                                        balancer

  • edmarnj

    1 Message

    2457

    0

    Posted November 17th, 2020 08:00

    We used the CAS API to delete all the CLIP-IDs in our ECS bucket, however over 17,000 objects still remain.  Is there a similar tool to the bucket-wipe JAR that takes a PEA file?  That is only method we have to connect to the bucket.

    Thank you,



  • DELL-Sam L

    Community Manager

    •

    8108 Posts

    •

    34586 Points

    2447

    0

    Posted November 17th, 2020 15:00

    Hello edmarnj,

    Here is a link to a coulpe of KB’s that maybe of assistance.

    https://dell.to/32UsmD9

    https://dell.to/3pz47UV