AllDup painfully slow

English support for the software AllDup
Post Reply
giovariot
Posts: 5
Joined: 24 Mar 2020, 21:34

AllDup painfully slow

Post by giovariot »

I'm using AllDup 4.5.5

I checked the duplicates between two hard disk backups, so admitedly around 900k files, so I expected it to take a while to analyze the whole bunch. It took a day and a half with the only filters activated being "same name" and "same file content" using sha1 checksum. And that's on around 400GB of files in total on a 100MB/s external hard drive. I expected a lot but this was a lot "lot".

After the duplicate analysis though I expected the program to work normally, this wasn't the case, though: there are around 200k duplicate groups, so quite a huge tree. The first command I gave was "expand all groups", I expected it to take a few minutes, it took me more than 2 hours.

When I asked alldup to select all the duplicates in one of the root folders it took another 2 hours and selected exactly the opposite file of those I wanted to select. Luckily the "invert selection" feature only took a couple of minutes.

Then I tried deleting the 160k selected files: in 6 hours it only deleted around 6000 files.

This happened in a virtual machine on a i7 processor with 8gb of dedicated RAM. So then, desperate, I saved the results and trasferred the AppData folder to my desktop computer, saving the 560MB asr4a file took more than an hour on my virtual machine.
So then I connected the hard drive to my desktop pc and transferred the AppData folder, it took me around 5s seconds. So hopeful I opened up AllDup and lodaded the results.

It's been loading the saved results for almost 2 hours, the windows title states: "0:23:58 / 82% / 197424/241540 / 1:47:05" on a 6th gen i5 with 16GB of ram and data saved on an SSD...

Is this normal? I didn't remember AllDup to be so slow. This feels unuseably slow :( Every other duplicate finder was "slow-ish" on this check, but none took more than a few hours to checksum all the files and none took hours to automatically select results from the results page.
therube
Posts: 322
Joined: 07 Nov 2012, 00:28

Re: AllDup painfully slow

Post by therube »

Do you have your Log file?
Might help in posting that.

Might your antivirus be interfering (i.e., scanning 900k files) ;-)?

Based on your settings, the scan would be doing a name+size comparison first - before any hashing kicks in & you would think that should be relatively quick.

Do you need to scan all 900k at once, or might it be better to scan only particular sets of directory trees at one time, then another set of trees...?

---

(Just something I ran...)
File count: 4072, is correct.
In this case, there are 2120 dup'd file sizes (& separately, 2033 dup'd file names).
Now if you take out the not name dup's, like RCORES*.dat, that's one off the list that it needs to run a hash on...
I don't know dup'd name+size, specifically, but AllDup's Checksums: 2186 could be reasonable for this data set.
.
AllDup compares up with Everything.png
giovariot
Posts: 5
Joined: 24 Mar 2020, 21:34

Re: AllDup painfully slow

Post by giovariot »

Hello, sorry this is in italian. I have largely exagerated the duration of the file scan, though. I really thought it took almost 2 days, probably because I went to sleep and it was still running. Apart from my exageration it took way more time than using other similar duplicates finding software, don't really know why.

Also deleting the files on the desktop "only" took 8 hours

Code: Select all

04/12/2021 11:39:01 - AllDup 4.5.5
04/12/2021 11:39:01 - Metodo di ricerca: Nome file + Contenuto file
04/12/2021 11:39:01 - Metodo di comparazione: Compara tutti i caratteri del nome file
04/12/2021 11:39:01 - Metodo di comparazione: MD5 (128-Bit)
04/12/2021 11:39:01 - Opzione: Use database
04/12/2021 11:39:01 - 1. Cartella origine: Z:\Samsung\windows
04/12/2021 11:39:01 - 2. Cartella origine: Z:\Samsung\windows43
04/12/2021 11:39:01 - Opzione: Compara file da tutte le cartelle sorgenti
04/12/2021 11:39:01 - Filtro Cartella attivato: 7
04/12/2021 11:39:01 - Tipologia filtro: Esclusivo
04/12/2021 11:39:01 - Cartella filtro 1: %3
04/12/2021 11:39:01 - Cartella filtro 2: %3
04/12/2021 11:39:01 - Cartella filtro 3: %3
04/12/2021 11:39:01 - Cartella filtro 4: %3
04/12/2021 11:39:01 - Cartella filtro 5: %3
04/12/2021 11:39:01 - Cartella filtro 6: %3
04/12/2021 11:39:01 - Cartella filtro 7: %3
04/12/2021 11:39:01 - Determina il numero di file di tutte le cartelle sorgenti...
04/12/2021 12:01:42 - Numero file: 957.916
04/12/2021 12:01:42 - Esaminati: Z:\Samsung\windows
04/12/2021 18:09:08 - File filtrati: 4.595
04/12/2021 18:09:08 - Esaminati: Z:\Samsung\windows43
04/12/2021 23:12:32 - File filtrati: 1.181
04/12/2021 23:12:36 - Trovati 427.081 duplicati con un totale di 77,69 GB all'interno della cartella 'Z:\Samsung\windows'
04/12/2021 23:12:36 - Trovati 198.179 duplicati con un totale di 10,19 GB all'interno della cartella 'Z:\Samsung\windows43'
04/12/2021 23:13:18 - File esaminati: 957.916
04/12/2021 23:13:18 - Gruppi: 241.540
04/12/2021 23:13:18 - Numero file comparati: 2.023.661
04/12/2021 23:13:18 - Checksum create: 729.879
04/12/2021 23:13:18 - Database checksums stored: 729.879
04/12/2021 23:13:18 - Duplicati: 625.260 (65%) (87,89 GB)
04/12/2021 23:13:18 - Tempo trascorso: 11:33:35
06/12/2021 16:43:07 - --------------------------------------------------
06/12/2021 16:43:07 - Azione 'Elimina file' iniziata.
06/12/2021 16:43:42 - Azione 'Elimina file' annullata.
06/12/2021 16:43:42 - 49 di 169641 file selezionati sono stati eliminati.
06/12/2021 16:43:42 - File rimossi dal risultato della ricerca: %2
06/12/2021 16:43:56 - --------------------------------------------------
06/12/2021 16:43:56 - Azione 'Elimina file' iniziata.
06/12/2021 17:41:00 - Azione 'Elimina file' annullata.
06/12/2021 17:41:00 - 1659 di 169592 file selezionati sono stati eliminati.
06/12/2021 17:41:00 - File rimossi dal risultato della ricerca: %2
06/12/2021 17:41:24 - --------------------------------------------------
06/12/2021 17:41:24 - Azione 'Sposta file nel Cestino' iniziata.
06/12/2021 17:45:18 - ERROR: Impossibile spostare il file 'Z:\Samsung\windows43\WINDOWS\WinSxS\Manifests\x86_netfx4-netfx_skuidentifier_full_4_8_b03f5f7f11d50a3a_4.0.15805.0_none_1599d413ca65fb8d.manifest' nel Cestino di Windows: Non esiste nessun Cestino di Windows sull'unità 'Z:\'!
06/12/2021 18:21:13 - ERROR: Impossibile spostare il file 'Z:\Samsung\windows43\Windows.old\WINDOWS\WinSxS\Manifests\x86_netfx4-netfx_skuidentifier_full_4_8_b03f5f7f11d50a3a_4.0.15788.0_none_13b65c37cd3cfd81.manifest' nel Cestino di Windows: Non esiste nessun Cestino di Windows sull'unità 'Z:\'!
06/12/2021 18:21:17 - Azione 'Sposta file nel Cestino' annullata.
06/12/2021 18:21:17 - 0 di 167933 file selezionati sono stati spostati nel cestino di Windows
06/12/2021 18:21:17 - Numero di errori: 2
06/12/2021 18:21:17 - File trascurati: 2
06/12/2021 18:21:38 - --------------------------------------------------
06/12/2021 18:21:38 - Azione 'Elimina file' iniziata.
06/12/2021 23:02:47 - Azione 'Elimina file' annullata.
06/12/2021 23:02:47 - 8163 di 167933 file selezionati sono stati eliminati.
06/12/2021 23:02:47 - File rimossi dal risultato della ricerca: %2
07/12/2021 04:25:30 - --------------------------------------------------
07/12/2021 04:25:30 - Azione 'Elimina file' iniziata.
07/12/2021 12:21:21 - Azione 'Elimina file' completa.
07/12/2021 12:21:21 - 159770 di 159770 file selezionati sono stati eliminati.
07/12/2021 12:21:21 - File rimossi dal risultato della ricerca: %2
Thanks for reply. Hope it helps. I can also upload somewhere the scan results..
Post Reply