"Private" Binary Data in ID3 Tag of MP3 Files causing Duplicate Matches to Fail ?

[jcramer]

For roughly the past 10 years, I have successfully used AllDup to accurately de-dup my large MP3 Music file collection.
However, now, when I run AllDup on my recently consolidated collection of all Music Files, from my Android Phone and 3 different Windows Computers, onto the Hard Disk Drives of 1 computer, AllDup 4.2, running on 64-bit Windows 10 Update 1809, is failing to identify many files which are duplicates.

I always use the "Ignore ID3 Tag" option and, in the past, that has always worked to identify duplicate files even if they have different ID3 Tag Values or different sized Album Art stored in the files.

So, I just used the Windows application "File Viewer Plus [Free]", from the Windows Store, to display the File Information from each of 2 files containing the same Music but were not identified as duplicates, and compared the File Properties & Tag Information of the two.
I checked that the "Length" of the music stored in each,as well as the Sample Rate and Audio Bitrate used to create the files, are exactly the same. I also listened to each Song File completely to make sure there are not aberrations in either. The embedded Album Art and ID3 sizes are different, but in the past that has not caused match failures.

HOWEVER, there is 1 difference between the pair of files. One contains the following information that the other does not:
"Private: (Binary data 8192 bytes, use -b option to extract)"

I do not know what the "Private Binary Data" stored in the one file represents. Both files were purchased from Amazon and, as is the case with all files purchased from the Amazon Music Store, do not have Digital Media Copy Protection and both contain, in the "Comment Field" of the ID3, the exact same Amazon Song identifier, for example:
"Comment: Amazon.com Song ID: 202593618"

Can the fact that one file has "Private Binary Data", and another identical Music Song File does not, be what is causing the 2 files to *not* be found as Duplicates by AllDup? If so, does anyone know what the procedure is, in Windows 10, to remove "Private Data" from files containing it, and if doing that would cause problems with playback of the music from the file or any other problem ?

Thanks in advance to anyone who can provide me with some guidance on this issue.
Jim

[Administrator]

Is it possible to send us two MP3-files that are identical but AllDup doesnt recognize them as duplicates because of the special binary data from amazon for testing/debugging purpose?

Some further information about the amazon privacy tag: https://aaronk.me/removing-personal-inf ... ff-amazon/

[jcramer]

Certainly, however each song is 2:20 Length , 256 bps, and has file size of 4,256 KB when 1 create a "Normally Compressed" ZIP file for it, and this Forum will not allow me to attach it because it is too large.

Is there another way that I can Send them (1 song per ZIP file) to you ?

[jcramer]

Also, in case it is of some help to you, I am going to Copy-Paste the very detailed info about each of the two files that Windows File Viewer Pro 3 gives me:

Here is the data for the file that HAS the Private Data in it:

File Information
----------------
Name: 01-05- The One That Got Away (Got Away With My Heart).mp3
File type: MP3 Audio File (.mp3)
MIME Type: audio/mpeg
Opens with: Nero MediaHome 2019
Size: 4 MB (4,750,674 bytes)
Location: M:\Seagate Slim Recovery -2 - ErasUS- Entire Disk\Recovered data 11-30 02_48_23\SG Slim 1TB J\MyMusic\MyMP3s\From Amazon\Allison Moorer\Alabama Song\
Created: March 25, 2017 6:26 PM
Modified: December 7, 2018 8:27 AM
Accessed: January 17, 2019 2:42 PM
Read-only: No
Hidden: No
System: No
MD5: 9defe11ecf63822887ebb4e36c52375f

Media Information
----------------
MPEG Audio Version: 1
Audio Layer: 3
Audio Bitrate: 256 kbps
Sample Rate: 44100
Channel Mode: Joint Stereo
MS Stereo: Off
Intensity Stereo: Off
Copyright Flag: False
Original Media: False
Emphasis: None
ID3 Size: 249254

Private: (Binary data 8192 bytes, use -b option to extract)

Title: The One That Got Away (Got Away With My Heart)
Album: Alabama Song
Genre: Country
Track: 5/11
Year: 1998
Band: Allison Moorer
Comment: Amazon.com Song ID: 202593621
Copyright: (C) 1998 MCA Records Nashville a division of MCA Records Inc.
Part Of Set: 1/1
Picture MIME Type: image/jpeg
Picture Type: Front Cover
Picture: (Binary data 235988 bytes, use -b option to extract)
WM Media Class Secondary ID: 00000000-0000-0000-0000-000000000000
WM Media Class Primary ID: D1607DBC-E323-4BE2-86A1-48A42A28441E
WM Provider: AMG
WM Content ID: FDB413EC-C46A-44CF-8796-AA9242BF7405
WM Collection Group ID: 1B896230-58AB-474F-997B-9AEC283631CF
Publisher: Universal Distribution
WM Unique File Identifier: (Binary data 114 bytes, use -b option to extract)
WM Collection ID: B5020207-474E-4720-879F-7E2EE60E8D00
Composer: Kostas Lazarides
Artist: Allison Moorer
Date/Time Original: 1998
Duration: 0:02:20 (approx)

FileInfo.com Information
.MP3 - 1 file type found
----------------
Type: MP3 Audio File
Category: Audio
Popularity: ★★★★★
Developer: Moving Picture Experts Group
--------

AND here is the data from a copy of the same Song/Music File which DOES NOT have Private Data in it.

File Information
----------------
Name: 05 - The One That Got Away (Got Away With My Heart).mp3
File type: MP3 Audio File (.mp3)
MIME Type: audio/mpeg
Opens with: Nero MediaHome 2019
Size: 4 MB (4,552,008 bytes)
Location: N:\Downloads\restore\Users\jcramer\Music\My MP3s\Amazon MP3\Allison Moorer\Alabama Song\
Created: December 5, 2010 5:32 PM
Modified: July 19, 2011 4:50 PM
Accessed: January 17, 2019 2:15 PM
Read-only: No
Hidden: No
System: No
MD5: bcbd694eb50bb216c5f2840aeb2f2b8b

Media Information
----------------
MPEG Audio Version: 1
Audio Layer: 3
Audio Bitrate: 256 kbps
Sample Rate: 44100
Channel Mode: Joint Stereo
MS Stereo: Off
Intensity Stereo: Off
Copyright Flag: False
Original Media: False
Emphasis: None
ID3 Size: 50460
Grouping: From Amazon MP3s
Title: The One That Got Away (Got Away With My Heart)
Copyright: (C) 1998 MCA Records Nashville a division of MCA Records Inc.
Genre: Country
Album: Alabama Song
Comment: Amazon.com Song ID: 202593621
Track: 5/11
Picture MIME Type: image/jpeg
Picture Type: Front Cover
Picture: (Binary data 45413 bytes, use -b option to extract)
Band: Allison Moorer
Part Of Set: 1/1
Year: 1998
Publisher: Universal Distribution
WM Media Class Secondary ID: 00000000-0000-0000-0000-000000000000
WM Media Class Primary ID: D1607DBC-E323-4BE2-86A1-48A42A28441E
WM Provider: AMG
WM Content ID: FDB413EC-C46A-44CF-8796-AA9242BF7405
WM Collection ID: 1B896230-58AB-474F-997B-9AEC283631CF
WM Collection Group ID: 1B896230-58AB-474F-997B-9AEC283631CF
WM Unique File Identifier: (Binary data 114 bytes, use -b option to extract)
Composer: Allison Moorer
Artist: Allison Moorer
Date/Time Original: 1998
Duration: 0:02:20 (approx)

FileInfo.com Information
.MP3 - 1 file type found
----------------
Type: MP3 Audio File
Category: Audio
Popularity: ★★★★★
Developer: Moving Picture Experts Group

If you find that the "Private Data", whatever that is, is what is causing the AllDup Match failures, then could the
Private Data be related to Windows. I have had many of the files which fail to Match as dups stored on Windows
XP, Vista, 7 and now 10 versions over 8+ years. The files with the Private Data might have been stored under
a certain Windows O.S. , while their corresponding duplicates which are failing to Match, might never have been
been stored on a machine with that same O.S.

Also, in my last Post I meant to thank you very much for your fast response. Much appreciated.
Jim

[Administrator]

Is there another way that I can Send them (1 song per ZIP file) to you ?

Please send each song via email to the email address "support@mtsd.de".

[jcramer]

The first of the 2 emails, each with 1 Music File attached, is being sent right now.
I know that some email servers will not send an attached zip file unless it is encrypted with a Password.

So, I am also sending the Password to unzip the files in a separate, 3rd email, now.

Jim

[Administrator]

some of your mp3 files are corrupted.
the ID3v1 tag exists twice at the end of the file.
please use the freeware "mp3diags" to scan your mp3 collection for corrupted files and repair them.