Slow Deletion

English support for the software AllDup
Post Reply
therube
Posts: 348
Joined: 07 Nov 2012, 00:28

Slow Deletion

Post by therube »

Oh, what to call this... Slow Deletion...
Perhaps expected, not sure?
(Not complaining. Just trying to get a better understanding.)

Code: Select all

02/19/2022 05:03:59 PM - INFO: Unable to detect VLC Media Player 32-bit version 3 on your system
02/19/2022 05:03:59 PM - --------------------------------------------------
02/19/2022 05:03:59 PM - AllDup 4.5.13 PE
02/19/2022 05:03:59 PM - Search method: File name + File size + File content
02/19/2022 05:03:59 PM - Comparison method: Compare all characters of a file name
02/19/2022 05:03:59 PM - Comparison method: SHA-1 (160-Bit)
02/19/2022 05:03:59 PM - Option: Use database
02/19/2022 05:03:59 PM - 1.Source folder: W:\T\+o._e-fat-e-ccc-ruben2002-2
02/19/2022 05:03:59 PM - 2.Source folder: W:\+o._e-fat-e-ccc-ruben2002-2
02/19/2022 05:03:59 PM - Option: Compare only files between different source folders
02/19/2022 05:03:59 PM - Folder filter activated: 7
02/19/2022 05:03:59 PM - Filter type: Exclusive
02/19/2022 05:03:59 PM - 1.folder filter: e:\windows
02/19/2022 05:03:59 PM - 2.folder filter: e:\program files (x86)
02/19/2022 05:03:59 PM - 3.folder filter: e:\program files
02/19/2022 05:03:59 PM - 4.folder filter: ?:\system volume information
02/19/2022 05:03:59 PM - 5.folder filter: ?:\recycled
02/19/2022 05:04:00 PM - 6.folder filter: ?:\recycler
02/19/2022 05:04:00 PM - 7.folder filter: ?:\$recycle.bin
02/19/2022 05:04:00 PM - Determine file count of all source folders...
02/19/2022 05:04:43 PM - File count: 422,693
02/19/2022 05:04:43 PM - Scan: W:\T\+o._e-fat-e-ccc-ruben2002-2
02/19/2022 05:04:46 PM - Files filtered: 4
02/19/2022 05:04:46 PM - Scan: W:\+o._e-fat-e-ccc-ruben2002-2
02/20/2022 06:51:06 AM - Files filtered: 4
02/20/2022 06:51:11 AM - Found 210,914 duplicates with a total of 12.95 GB inside folder 'W:\T\+o._e-fat-e-ccc-ruben2002-2'
02/20/2022 06:51:11 AM - Found 210,805 duplicates with a total of 12.94 GB inside folder 'W:\+o._e-fat-e-ccc-ruben2002-2'
02/20/2022 06:51:25 AM - Scanned files: 422,693
02/20/2022 06:51:25 AM - Groups: 206,924
02/20/2022 06:51:25 AM - File comparison count: 214,856
02/20/2022 06:51:25 AM - Checksums created: 421,719
02/20/2022 06:51:25 AM - Database checksums stored: 421,719
02/20/2022 06:51:25 AM - Duplicates: 421,719 (99%) (25.89 GB)
02/20/2022 06:51:25 AM - Elapsed time: 13:47:12
02/20/2022 07:57:07 AM - --------------------------------------------------
02/20/2022 07:57:07 AM - Action 'Delete files' started.
02/20/2022 10:23:59 AM - Action 'Delete files' complete.
02/20/2022 10:23:59 AM - Files deleted: 210805 of 210805
02/20/2022 10:23:59 AM - Groups removed from the search result: 203102
02/20/2022 10:23:59 AM - Files removed from the search result: 210805
I ran a search.
Paused it before it had finished.
Put the computer to sleep.
Then resumed the search at a later point.

On return, remaining search was relatively quick.

After that...

Expanding all the "folders" took a LONG time

Selecting all the wanted files for deletion, was quicker

Actually (hard) deletion; 210805 files, 12 GB, took --- 2:26:52
--- A LONG TIME


- granted, this was on a VERY slow 30kbps external USB2.0 hdd

Why the extended deletion time?
Due to the actual deletions, or due to the updating of the .db (sha1 hashes),
or perhaps due to the fact that all the files were small in size (or my slug of a drive)?

- /if/ due to the .db (.adb), then maybe there is a more efficient way to go about that?
Perhaps a separate "purge" routine for the .db (after the fact) that can go out &
check against non-existing files (on disk), mark those, then purge those (from the .db) en mass ?

Just how, when is the .db updated?
As in, is my assumption that you do that as the deletion routine runs correct?


---


Note that the above shown "Elapsed time:" is not real (cause the computer was put to sleep).
(I'll post more on that, another day.)
Administrator
Site Admin
Posts: 4149
Joined: 04 Oct 2004, 18:38
Location: Thailand
Contact:

Re: Slow Deletion

Post by Administrator »

"Deleting files" doesnt touch/update the DB.
I guess the most time will be used to update the search result grid.
These grids are getting slower and slower with a large amount of files.
Deactivate the options "Remove groups with only one file", "Remove files from the search result" and activate "Don't log any file actions (faster)" before you delete all files to speed up the process.
undupit
Posts: 19
Joined: 16 Nov 2024, 05:26

Re: Slow Deletion

Post by undupit »

Administrator wrote: 02 Mar 2022, 07:40 These grids are getting slower and slower with a large amount of files.
The deleting is not twice as slow if you delete twice as many files. It's more like 4 times as slow.
Under 10000 files, the deleting is doable. Above 10000, the deleting performance it's rapidly degrading. Something like parabolically slower.
Deleting times, roughly:
Under 7000 files: 'few' seconds.
14000 files: 1m20s
28000 files: 7m30s
56000 files: 47m00s
80000 files: 1h47m30s
You see, the time it takes to delete more files, is not linear at all.
I hope you can change the code, so that it's a bit more like linear performance, not parabolically.

For now, I have to cancel the AllDup search halfway over and over again, if there are a lot of dupes, because the deletion takes a huge amount of time if the number of dupes go up.
Then I have to redo and redo and redo the same search, which again and again and again needs to 'enlist' all files before the compare starts, which was already done in the first run, but has to be done again and again, doing the same thing over and over again, it doesn't remember what it did in the first run.
You can Imagine how hard it is to use AllDup to delete millions of files, if you compare two backups, with over millions of dupes.
Administrator
Site Admin
Posts: 4149
Joined: 04 Oct 2004, 18:38
Location: Thailand
Contact:

Re: Slow Deletion

Post by Administrator »

Have you tried:
Administrator wrote: 02 Mar 2022, 07:40 Deactivate the options "Remove groups with only one file", "Remove files from the search result" and activate "Don't log any file actions (faster)" before you delete all files to speed up the process.
These options exclude the time to update the grid and the logging.
undupit
Posts: 19
Joined: 16 Nov 2024, 05:26

Re: Slow Deletion

Post by undupit »

Administrator wrote: 24 Jan 2026, 11:52 Have you tried:
Administrator wrote: 02 Mar 2022, 07:40 Deactivate the options "Remove groups with only one file", "Remove files from the search result" and activate "Don't log any file actions (faster)" before you delete all files to speed up the process.
These options exclude the time to update the grid and the logging.
Those things have as good as no influence on the parabolic slowdown if you delete many files.
Administrator
Site Admin
Posts: 4149
Joined: 04 Oct 2004, 18:38
Location: Thailand
Contact:

Re: Slow Deletion

Post by Administrator »

The slow data deletion must be related to the external hard drive you are using.

In a test, we deleted 100,000 files from the search results onto an external USB 3.0 hard drive. The process took only 2 minutes. See log:

Code: Select all

28.01.2026 13:34:39 - AllDup 4.5.72
28.01.2026 13:34:39 - Search method: File size
28.01.2026 13:34:39 - 1.Source folder: D:\!test
28.01.2026 13:34:39 - Option: Compare files from all source folders
28.01.2026 13:34:39 - Determine file count of all source folders...
28.01.2026 13:34:39 - File count: 100.000
28.01.2026 13:34:39 - Scan: D:\!test
28.01.2026 13:34:49 - Found 100.000 duplicates with a total of 1,48 MB inside folder 'D:\!test'
28.01.2026 13:34:50 - Scanned files: 100.000
28.01.2026 13:34:50 - Groups: 10
28.01.2026 13:34:50 - File comparisons: 100.000
28.01.2026 13:34:50 - Duplicates: 100.000 (100%) (1,48 MB)
28.01.2026 13:34:50 - Elapsed time: 00:00:10
28.01.2026 13:36:13 - --------------------------------------------------
28.01.2026 13:36:13 - Action 'Delete files' started.
28.01.2026 13:36:14 - DELETED: D:\!test\Folder01\010000.txt
28.01.2026 13:36:14 - DELETED: D:\!test\Folder01\009999.txt
....
28.01.2026 13:38:15 - DELETED: D:\!test\Folder10\090001.txt
28.01.2026 13:38:15 - Action 'Delete files' complete.
28.01.2026 13:38:15 - Files deleted: 100000 of 100000
28.01.2026 13:38:15 - Groups with all files selected: 10
28.01.2026 13:38:15 - Groups removed from the search result: 10
28.01.2026 13:38:15 - Files removed from the search result: 100000
test system: Win10-laptop, i7-12700H, 64GB
therube
Posts: 348
Joined: 07 Nov 2012, 00:28

Re: Slow Deletion

Post by therube »

200K files & I'm looking at ~18 minutes (or maybe ~15 minutes).
Win7, i7-3770S, 8 GB RAM, 1 TB SSD (runs acceptably for my needs, but dated, old)
(first time, I thought I'd deselected the... (something about all files selected option), but maybe i missed it?)

Duplicates based on size.
After found, Select All, then set to Permanently Delete.

Code: Select all

02/11/2026 11:59:16 AM - AllDup 4.5.72 PE
02/11/2026 11:59:16 AM - Search method: File size
02/11/2026 11:59:16 AM - 1.Source folder: C:\out\100k-2 - Copy (2)
02/11/2026 11:59:16 AM - Option: Compare files from all source folders
02/11/2026 11:59:16 AM - Folder filter activated: 7
02/11/2026 11:59:16 AM - Filter type: Exclusive
02/11/2026 11:59:16 AM - 1.folder filter: e:\windows
02/11/2026 11:59:16 AM - 2.folder filter: e:\program files (x86)
02/11/2026 11:59:16 AM - 3.folder filter: e:\program files
02/11/2026 11:59:16 AM - 4.folder filter: ?:\system volume information
02/11/2026 11:59:16 AM - 5.folder filter: ?:\recycled
02/11/2026 11:59:16 AM - 6.folder filter: ?:\recycler
02/11/2026 11:59:16 AM - 7.folder filter: ?:\$recycle.bin
02/11/2026 11:59:16 AM - Determine file count of all source folders...
02/11/2026 11:59:21 AM - File count: 204,777
02/11/2026 11:59:21 AM - Scan: C:\out\100k-2 - Copy (2)
02/11/2026 12:00:16 PM - Found 204,777 duplicates with a total of 19.89 MB inside folder 'C:\out\100k-2 - Copy (2)'
02/11/2026 12:00:36 PM - Scanned files: 204,777
02/11/2026 12:00:36 PM - Groups: 15
02/11/2026 12:00:36 PM - File comparisons: 204,777
02/11/2026 12:00:36 PM - Duplicates: 204,777 (100%) (19.89 MB)
02/11/2026 12:00:36 PM - Elapsed time: 00:01:00
02/11/2026 12:01:35 PM - --------------------------------------------------
02/11/2026 12:01:35 PM - Action 'Delete files' started.
02/11/2026 12:01:39 PM - Action 'Delete files' complete.
02/11/2026 12:01:39 PM - Files deleted: 0 of 204777
02/11/2026 12:01:39 PM - Skipped files: 204777
02/11/2026 12:01:39 PM - Groups protected: 15 (files: 204777)
02/11/2026 12:01:39 PM - INFO: The 'Do not process groups with all files selected' option is enabled, therefore files in such groups were not processed.
02/11/2026 12:01:39 PM - Files protected: 204777
02/11/2026 12:03:00 PM - --------------------------------------------------
02/11/2026 12:03:00 PM - Action 'Delete files' started.
02/11/2026 12:17:26 PM - Action 'Delete files' complete.
02/11/2026 12:17:26 PM - Files deleted: 204777 of 204777
02/11/2026 12:17:26 PM - Groups with all files selected: 15
02/11/2026 12:17:26 PM - Groups removed from the search result: 15
02/11/2026 12:17:26 PM - Files removed from the search result: 204777

Copying 200K files takes ~5 mintes.

Code: Select all

C:\out>fcp 100k-2 /to=100k-222
=================================================
fcp(ver5.11.2) start at 2026/02/11 12:28:17
<Source>  C:\out\100k-2
<DestDir> C:\out\100k-222
<Command> Diff (Size/Date)
<FileLog> C:\BIN\Log\20260211-122817-0.log
-------------------------------------------------

TotalRead  = 19 MiB
TotalWrite = 19 MiB
TotalFiles = 204,777 (3)
TotalTime  = 04:26
TransRate  = 0.08 MB/s
FileRate   = 768.1 files/s

Result : (ErrFiles : 0 / ErrDirs : 0) at 2026/02/11 12:32:43
(Hard) deleting 200K - through my file manager, took ~50 seconds.

Code: Select all

TimeThis :  Command Line :  pause
TimeThis :    Start Time :  Wed Feb 11 12:37:30 2026
TimeThis :      End Time :  Wed Feb 11 12:38:19 2026
TimeThis :  Elapsed Time :  00:00:49.669
Another slow delete thread (where the time to perform the deletes didn't seem to make sense - to me), Making sure deleting works.
Last edited by therube on 12 Feb 2026, 00:46, edited 3 times in total.
therube
Posts: 348
Joined: 07 Nov 2012, 00:28

Re: Slow Deletion

Post by therube »

@undupit, might you be using a ReFS formatted drive (Win11) ?
Administrator
Site Admin
Posts: 4149
Joined: 04 Oct 2004, 18:38
Location: Thailand
Contact:

Re: Slow Deletion

Post by Administrator »

Perhaps it's the antivirus software being used that's slowing down the deletion process?
Or maybe it's the combination of antivirus software, operating system, and hard drive hardware...
Post Reply