Deleting Lots of (Small?) Files Seems Inefficient

[therube]

Deletion of Lots of (Small?) Files Seems to be Inefficient ?

80,000 files, all < 10 KB, & I'm at 50% complete (deletion, direct) & running at the 11:min mark.
(Deletion finished at 15:min.)

Single top level directory.
100 subdirectories.
All files are within a single second-level directory of the 100 subdirectories.

Code: Select all

C:/OUT
C:/OUT/2007.0911
C:/OUT/2007.0911/TMP
C:/OUT/2007.0911/TMP/file1, file2, file3 ...

[Administrator]

What version (number) of AllDup do you use?

[therube]

Oops, sorry about that, should have mentioned that.
3.9.15
And I guess I should have put this in the Beta thread too.

[therube]

Just a FWIW (& I don't know if I was logging file actions or not during my earlier post) ...

3.9.18

I had unchecked, some time ago & didn't realize, Don't log any file actions.
Probably should not have done that - in this case .
(Resulted in a 35 MB log file, & I'm sure that also severly increased the time it took to delete the files.)

Code: Select all

03/03/2016 10:24:26 AM - --------------------------------------------------
03/03/2016 10:24:26 AM - Search: File name + File size + File content SHA-1 (160-Bit)
03/03/2016 10:24:26 AM - Determine the file count of all source folders...
03/03/2016 10:27:16 AM - File Count: 140711
03/03/2016 10:27:16 AM - Scan: X:\out\0neil
03/03/2016 11:23:11 AM - Found 140530 duplicates with 15,014,364,291 Bytes in source folder 'X:\out\0neil'
03/03/2016 11:23:13 AM - Groups: 1,998
03/03/2016 11:23:13 AM - File Comparison Count: 141,181
03/03/2016 11:23:13 AM - Duplicates: 140530 (99%) (13.98 GB)
03/03/2016 11:23:13 AM - Elapsed time: 00:58:47
03/03/2016 11:40:26 AM - --------------------------------------------------
03/03/2016 11:40:26 AM - Action 'Delete files' started.
...
03/03/2016 12:23:49 PM - Action 'Delete files' finished.
138532 files(s) have been processed.

[sdfgdhfgh]

Well, did you really delete or move to recycler?
The first one is faster.
This process is slow, because every single file has to be deleted/moved.
I don't know whether AllDup detects when there is a folder containing only files which are to be deleted, but i dont think so.

If all files of a specific directory are to be deleted, the performance could probably be improved by deleting the directory instead of every single file.

If not all files are deleted, but most of a large number of files, it might be faster to move the not-to-be-deleted-files to another folder, remove that folder and rename the new folder to the name of the old folder, maybe including timestamps. However there would be a problem if this process crashes without having a recovery-log.