Highlighted
2 Bronze

Delete large bucket

If I have a bucket with a very large number of files, what is the best way to recursively delete the objects so I can delete the bucket?

With s3cmd, it's so large recursive delete fails. For one of the smaller buckets I'm doing s3cmd ls + s3cmd del using gnu parallel. But the larger bucket is 100,000,000+ objects and this method doesn't work.

0 Kudos
3 Replies
Highlighted
Anonymous
Not applicable

Re: Delete large bucket

There is an internal "Bucket Wipe" tool that should work if anything does, you can find a link at Ken Steinfeldt's page: https://inside.dell.com/docs/DOC-333834. It spins off multiple delete threads.

0 Kudos
Highlighted
2 Bronze

Re: Delete large bucket

I can’t view that page as it says dell internal only.

If you are an External user and require access to inside Dell then please contact your Dell representative for more information. Please note External User access is limited to Dell approved customers, partners, and suppliers.

--

John Daniels

Research IS Systems Engineer III

Children's Hospital of Philadelphia Research Institute

The Roberts Center for Pediatric Research

2716 South Street

Philadelphia, PA 19104

danielsj1@email.chop.edu<mailto:danielsj1@email.chop.edu>

Office: 267-425-0155

Need faster service?

Try placing your request at http://cirrus.research.chop.edu<http://cirrus.research.chop.edu/> before opening a manual service request.

Otherwise, you can submit an adhoc request here<https://chop.service-now.com/esp?sysparm_cancelable=true> (general->other), include ‘please route to research IS’

0 Kudos
Highlighted
3 Argentum

Re: Delete large bucket

You can get the bucket-wipe tool here:

WARNING: This will erase the bucket and all of its data!  Please make absolutely sure this is what you want.

http://130753149435015067.public.ecstestdrive.com/share/bucket-wipe-1.9.jar


usage: java -jar bucket-wipe.jar [options] <bucket-name>

-a,--access-key                    the S3 access key

-e,--endpoint                      the endpoint to connect to, including

                                    protocol, host, and port

-h,--help                          displays this help text

-hier,--hierarchical               Enumerate the bucket hierarchically.  This

                                    is recommended for ECS's

                                    filesystem-enabled buckets.

    --keep-bucket                   do not delete the bucket when done

-l,--key-list                      instead of listing bucket, delete objects

                                    matched in source file key list

    --no-smart-client               disables the ECS smart-client. use this

                                    option with an external load balancer

-p,--prefix                        deletes only objects under the specified

                                    prefix

-s,--secret-key                    the secret key

    --stacktrace                    displays full stack trace of errors

-t,--threads                       number of threads to use

    --vhost                         enables DNS buckets and turns off load

                                    balancer

0 Kudos