AllDup painfully slow
Posted: 07 Dec 2021, 07:38
I'm using AllDup 4.5.5
I checked the duplicates between two hard disk backups, so admitedly around 900k files, so I expected it to take a while to analyze the whole bunch. It took a day and a half with the only filters activated being "same name" and "same file content" using sha1 checksum. And that's on around 400GB of files in total on a 100MB/s external hard drive. I expected a lot but this was a lot "lot".
After the duplicate analysis though I expected the program to work normally, this wasn't the case, though: there are around 200k duplicate groups, so quite a huge tree. The first command I gave was "expand all groups", I expected it to take a few minutes, it took me more than 2 hours.
When I asked alldup to select all the duplicates in one of the root folders it took another 2 hours and selected exactly the opposite file of those I wanted to select. Luckily the "invert selection" feature only took a couple of minutes.
Then I tried deleting the 160k selected files: in 6 hours it only deleted around 6000 files.
This happened in a virtual machine on a i7 processor with 8gb of dedicated RAM. So then, desperate, I saved the results and trasferred the AppData folder to my desktop computer, saving the 560MB asr4a file took more than an hour on my virtual machine.
So then I connected the hard drive to my desktop pc and transferred the AppData folder, it took me around 5s seconds. So hopeful I opened up AllDup and lodaded the results.
It's been loading the saved results for almost 2 hours, the windows title states: "0:23:58 / 82% / 197424/241540 / 1:47:05" on a 6th gen i5 with 16GB of ram and data saved on an SSD...
Is this normal? I didn't remember AllDup to be so slow. This feels unuseably slow Every other duplicate finder was "slow-ish" on this check, but none took more than a few hours to checksum all the files and none took hours to automatically select results from the results page.
I checked the duplicates between two hard disk backups, so admitedly around 900k files, so I expected it to take a while to analyze the whole bunch. It took a day and a half with the only filters activated being "same name" and "same file content" using sha1 checksum. And that's on around 400GB of files in total on a 100MB/s external hard drive. I expected a lot but this was a lot "lot".
After the duplicate analysis though I expected the program to work normally, this wasn't the case, though: there are around 200k duplicate groups, so quite a huge tree. The first command I gave was "expand all groups", I expected it to take a few minutes, it took me more than 2 hours.
When I asked alldup to select all the duplicates in one of the root folders it took another 2 hours and selected exactly the opposite file of those I wanted to select. Luckily the "invert selection" feature only took a couple of minutes.
Then I tried deleting the 160k selected files: in 6 hours it only deleted around 6000 files.
This happened in a virtual machine on a i7 processor with 8gb of dedicated RAM. So then, desperate, I saved the results and trasferred the AppData folder to my desktop computer, saving the 560MB asr4a file took more than an hour on my virtual machine.
So then I connected the hard drive to my desktop pc and transferred the AppData folder, it took me around 5s seconds. So hopeful I opened up AllDup and lodaded the results.
It's been loading the saved results for almost 2 hours, the windows title states: "0:23:58 / 82% / 197424/241540 / 1:47:05" on a 6th gen i5 with 16GB of ram and data saved on an SSD...
Is this normal? I didn't remember AllDup to be so slow. This feels unuseably slow Every other duplicate finder was "slow-ish" on this check, but none took more than a few hours to checksum all the files and none took hours to automatically select results from the results page.