How to delete millions of small files from Celerra?

no - the control station is just a NFS client as well and typically slower than your real NFS clients

unless you want to delete the while file system there is no other way than a rm

of course deleting the directory itself with like rm -rf is going to be faster than a find | rm

deleting a file is an expensive operation - it needs to

- update the directory

- delete the inode

- unlink the data block(s)

- update the block and inode free lists

Rainer

To second Rainer's comments, I'd suggest using an NFS client other than the control station, something that talks to the Celerra over it's public network interfaces.  I do this from an external desktop and it's much, much faster.  Instead of doing the delete in the same command, I'd run a short script with mtime to populate a file with all the filenames of the old files you wish to delete.  Then I'd run several commands in parallel to match the names and delete the files.  It should be much, much faster.   The key is to not try and do it over the backend network - it's just too painful.

Thanks!

Karl

Hello Rainer,

Thanks for the suggestion. Unfortunately I cannot delete the whole directory - i need to delete files which are 5 days or older. Simply running rm -rf doesn't work - it cannot handle that many files, that's why I have to use find with either -delete (fastest), xargs rm (faster) and -execute rm {} (slow).

Thx, Serge.

sounds like a good plan

you want to keep both the client work and the network traffic to a minimum

I'm not surprised that -exec is the slowest - forking a new rm for each delete is an expensive process

try multiple processes, good NFS client settings, see if maybe NFSv4 helps

of course it would be even better if you could teach your app to use multiple directories in the first place :-)