Unsolved
1 Rookie
•
37 Posts
0
65
August 6th, 2024 14:52
Solution for large scale autodelete
We are a national public broadcaster that produce a lot of video, which is stored on powerscale.
Most of this is only relevant for a short time, then it needs to be deleted.
We use a pretty advanced bash script on a server that has dozens of volumes exported from the powerscale clusters via nfs.
For scale there is some 14 million folders and tens of PB of data.
Anyhow it's become a pain to maintain this script, the guy who wrote it left almost a decade ago. There are literally thousands of different settings for thousands of folders. Yet we're running into more and more limitations of what it can and cannot do.
So... we're looking for some "autodelete"/file management product that can be bought and supported by and established vendor. So far haven't found any...
Suggestions?
storageSysAdmin
1 Rookie
•
58 Posts
0
August 12th, 2024 12:06
I would suggest just running the isi treedelete cmd on the top of the directory tree you wish to delete, its extremely efficient. but note, you cannot stop it once it starts, so tripple check your /path/to/dir
mawi82
1 Rookie
•
37 Posts
0
August 16th, 2024 07:38
@storageSysAdmin maybe the question wasn't clear but it's not about a command(I know about treedelete, thanks). It's about some sort of automation platform that can check age of files and act on them, create reports of what it's deleted, have a multitude of configuration options etc.
Im thinking more and more there isn't anything out there, even though lots of media & entertainment companies must have the same issue. I guess most(like us) develop their own solutions for this. So I will send a wish list to our developers I suppose.
(edited)
storageSysAdmin
1 Rookie
•
58 Posts
0
August 19th, 2024 16:11
@mawi82 ah yes I think we all have that issue these days. If you do come across something I would like to know what it is. However you are right that you pretty much would have to use a large 3rd party tool, e.g. Komprise is good for archiving based on ages etc.. but dont know if it actually purges.