Unsolved
3 Posts
0
5424
Delete large bucket
If I have a bucket with a very large number of files, what is the best way to recursively delete the objects so I can delete the bucket?
With s3cmd, it's so large recursive delete fails. For one of the smaller buckets I'm doing s3cmd ls + s3cmd del using gnu parallel. But the larger bucket is 100,000,000+ objects and this method doesn't work.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
October 23rd, 2018 11:00
There is an internal "Bucket Wipe" tool that should work if anything does, you can find a link at Ken Steinfeldt's page: https://inside.dell.com/docs/DOC-333834. It spins off multiple delete threads.
danielsj1
3 Posts
0
October 23rd, 2018 12:00
I can’t view that page as it says dell internal only.
“
If you are an External user and require access to inside Dell then please contact your Dell representative for more information. Please note External User access is limited to Dell approved customers, partners, and suppliers.
“
--
John Daniels
Research IS Systems Engineer III
Children's Hospital of Philadelphia Research Institute
The Roberts Center for Pediatric Research
2716 South Street
Philadelphia, PA 19104
danielsj1@email.chop.edu
Office: 267-425-0155
Need faster service?
Try placing your request at http://cirrus.research.chop.edu before opening a manual service request.
Otherwise, you can submit an adhoc request here (general->other), include ‘please route to research IS’
chris_arnett
110 Posts
0
October 25th, 2018 13:00
You can get the bucket-wipe tool here:
WARNING: This will erase the bucket and all of its data! Please make absolutely sure this is what you want.
http://130753149435015067.public.ecstestdrive.com/share/bucket-wipe-1.9.jar
usage: java -jar bucket-wipe.jar [options]-a,--access-key the S3 access key-e,--endpoint the endpoint to connect to, includingprotocol, host, and port-h,--help displays this help text-hier,--hierarchical Enumerate the bucket hierarchically. Thisis recommended for ECS'sfilesystem-enabled buckets.--keep-bucket do not delete the bucket when done-l,--key-list instead of listing bucket, delete objectsmatched in source file key list--no-smart-client disables the ECS smart-client. use thisoption with an external load balancer-p,--prefix deletes only objects under the specifiedprefix-s,--secret-key the secret key--stacktrace displays full stack trace of errors-t,--threads number of threads to use--vhost enables DNS buckets and turns off loadbalanceredmarnj
1 Message
0
November 17th, 2020 08:00
We used the CAS API to delete all the CLIP-IDs in our ECS bucket, however over 17,000 objects still remain. Is there a similar tool to the bucket-wipe JAR that takes a PEA file? That is only method we have to connect to the bucket.
Thank you,
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
November 17th, 2020 15:00
Hello edmarnj,
Here is a link to a coulpe of KB’s that maybe of assistance.
https://dell.to/32UsmD9
https://dell.to/3pz47UV