Page 1 of 1
Deleting Lots of (Small?) Files Seems Inefficient
Posted: 27 Jan 2016, 02:06
by therube
Deletion of Lots of (Small?) Files Seems to be Inefficient ?
80,000 files, all < 10 KB, & I'm at 50% complete (deletion, direct) & running at the 11:min mark.
(Deletion finished at 15:min.)
Single top level directory.
100 subdirectories.
All files are within a single second-level directory of the 100 subdirectories.
Code: Select all
C:/OUT
C:/OUT/2007.0911
C:/OUT/2007.0911/TMP
C:/OUT/2007.0911/TMP/file1, file2, file3 ...
Re: Deleting Lots of (Small?) Files Seems Inefficient
Posted: 27 Jan 2016, 19:15
by Administrator
What version (number) of AllDup do you use?
Re: Deleting Lots of (Small?) Files Seems Inefficient
Posted: 27 Jan 2016, 23:43
by therube
Oops, sorry about that, should have mentioned that.
3.9.15
And I guess I should have put this in the Beta thread too.
Re: Deleting Lots of (Small?) Files Seems Inefficient
Posted: 04 Mar 2016, 00:42
by therube
Just a FWIW (& I don't know if I was logging file actions or not during my earlier post) ...
3.9.18
I had unchecked, some time ago & didn't realize, Don't log any file actions.
Probably should not have done that - in this case
.
(Resulted in a 35 MB log file, & I'm sure that also severly increased the time it took to delete the files.)
Code: Select all
03/03/2016 10:24:26 AM - --------------------------------------------------
03/03/2016 10:24:26 AM - Search: File name + File size + File content SHA-1 (160-Bit)
03/03/2016 10:24:26 AM - Determine the file count of all source folders...
03/03/2016 10:27:16 AM - File Count: 140711
03/03/2016 10:27:16 AM - Scan: X:\out\0neil
03/03/2016 11:23:11 AM - Found 140530 duplicates with 15,014,364,291 Bytes in source folder 'X:\out\0neil'
03/03/2016 11:23:13 AM - Groups: 1,998
03/03/2016 11:23:13 AM - File Comparison Count: 141,181
03/03/2016 11:23:13 AM - Duplicates: 140530 (99%) (13.98 GB)
03/03/2016 11:23:13 AM - Elapsed time: 00:58:47
03/03/2016 11:40:26 AM - --------------------------------------------------
03/03/2016 11:40:26 AM - Action 'Delete files' started.
...
03/03/2016 12:23:49 PM - Action 'Delete files' finished.
138532 files(s) have been processed.
Re: Deleting Lots of (Small?) Files Seems Inefficient
Posted: 03 Aug 2016, 16:19
by sdfgdhfgh
Well, did you really delete or move to recycler?
The first one is faster.
This process is slow, because every single file has to be deleted/moved.
I don't know whether AllDup detects when there is a folder containing only files which are to be deleted, but i dont think so.
If all files of a specific directory are to be deleted, the performance could probably be improved by deleting the directory instead of every single file.
If not all files are deleted, but most of a large number of files, it might be faster to move the not-to-be-deleted-files to another folder, remove that folder and rename the new folder to the name of the old folder, maybe including timestamps. However there would be a problem if this process crashes without having a recovery-log.