Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

' Create a string representing the 64 bit persistent ID of an iTunes object

' Modified 2012-08-24

Function PersistentID(T)

PersistentID=Right("0000000" & Hex(iTunes.ITObjectPersistentIDHigh(T)),8) & "-" & Right("0000000" & Hex(iTunes.ITObjectPersistentIDLow(T)),8)

End Function

Also, the script is just replacing/ removing the entries in the itl, can't you just delete the itl and generate a new one with the steps so that you don't have all that extra code? Did you write this tt2? If you did I would think you know about the encoding of the database string from the script. The playlists are separate files are they not? You can export them and re-import them? Let me know what you think.

A random collection of bits of information about iTunes. This is by no means everything that is germane to this topic.

• Internally an iTunes playlist is an ordered list of library persistent IDs. Changing metadata doesn't affect the IDs, though it can affect which tracks qualify for membership of smart playlists.
• An iTunes library can include multiple playlists, smart playlists and playlist folders. All of the lists and their structure is stored internally. Lists can be exported individually, but they don't ordinarily exist as separate files.
• An exported playlist is an ordered list of file paths. Changing metadata may affect where a file is stored, so an exported playlist cannot always be reimported without changes.
• Smart playlist rule sets are only stored in the .itl file. You can export a list, and reimport as a regular playlist, but you cannot export and then reimport the rules.
• When an Apple device is synced with the library, iTunes can compare the contents of the device in terms of persistent IDs with the selection of content in the library that is due to sync, and copy over only the new or changed media, and/or remove files that no longer qualify for the device. The IDs are also used to sync ratings, play and skip counts, and last played/skipped times.
• If you create a new library, even with the same media, each track will be assigned a new ID, breaking down the relationship between IDs on any device and those in the library.
• Each iTunes library has an internal ID which, among other things, iTunes uses to recognize when a device is connected to a new library. In such cases iTunes will normally want to remove existing content on the device and repopulate from the current library. This probably harks back in part to Apple's attempts to limit casual piracy but is also necessary because the IDs on the device won't match those in the library.
• Certain metadata including, but not limited to, data added, checked, rating, play and skip counts and times, and all metadata for .wav files is only stored in the .itl file. Creating a new library file causes all of this information to be lost.

Destroying the current .itl file and reimporting the media has unwanted side effects. Obviously some users may not care about the loss of the information, but for those that do it matters.

The script, which is indeed my work, makes a number of passes through the selected tracks identifying different types of duplicate and thins them out in a way that tries to preserve as much information as possible. For example if there are two copies of a song one of which has been added to playlist A, and another to playlist B, then after deduping the remaining copy of the song will be in both playlists. Play & skip counts are merged, with the remaining track getting the most recent play and/or skip time. Where a track exists in two different formats/bitrates the script will try to preserve the higher quality one, but the user can opt to favour the smaller, or one format over another. This also works even if there are 10s or 100s of playlists involved, and any number of duplicates of the same track. Where practical the script tries to preserve the earliest date added value. When duplicate files are deleted or archived, and the last media file is removed from a folder, then the script can also clean up any artwork or desktop.ini files that might have been generated by Windows Media Player and remove the empty folder. The script uses the same criteria as iTunes does for the Show Exact Duplicates feature. This allows different versions of the same song on different albums, or by different artists to be kept in the library. Working solely with filenames could easily get this wrong. The script is as complicated as it needs to be to work the way I think it needs to, and to accommodate options that others have asked me to include. I wrote it not because I have any need of it, but to see if I could. As I've noted in how to delete duplicated songs all at one time? deduping is not a trivial problem.

tt2

Bro, that is FROM the script.... I think we are done here.... Use my simple solution or try to go through that mess you are suggesting, it is up to the user:

1. Backup your music to an external drive or another location on your physical computer drives.
2. If you already have double files, use a Duplicate file remover such as tt2's script DeDuper or another program such as Digital Volcano's Duplicate File Remover Pro. iTunes could, but never will make this feature. Note: Deduper will go into the iTunes files and the following steps will not be necessary after setting up the program inputs.
3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.
4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.
5. Close iTunes
6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes
7. Open iTunes and wait for everything to populate.
8. Change any settings you would like to use under preferences > advanced from then on, but follow steps 1 to 6 for new stuff.

I recognize my own code thanks. The script is just a program like any other. You run it, it does what it is supposed to do. Nobody needs to look inside it to make use of it. I thought you were actually interested in what is going on under the hood both with iTunes and the script. No problem if you aren't.

tt2

Your code does works great. iTunes also rebuilds the library for you, so you could just allow it to run and regenerate the library from all of the other files without needing to call the ID's from the database and count through the appendices to get the duplicates. It should just update the playlists since they are looking for links that won't exist anymore. A simple test would be to create a playlist with duplicate files, close iTunes, delete the itl., delete the physical file except for one, and then reopen iTunes. If it works you know you can delete it without fault. The information about the music file itself won't get lost because once iTunes opens it, either the info gets pulled from the file or from the web, so artwork and info isn't an issue either.

Now, you may be using the iTunes ID's to find duplicates, in which case if you're in the .itl already and you are already counting through the IDs then you might as well use what you are storing in those classes. If the script is doing that, then that makes more sense to me. That also means those aren't just a random collection of bits, they are your key to finding duplicates. If you are looking in the physical directories for dupes, then I think the simple way would be to allow iTunes to regenerate the itl, and playlist files.

I've tried to stress that as well as the standard tag data that is imported when tracks are added to a new empty library there is plenty of data that isn't; ratings, playlists, play & skip counts & dates, all metadata associated with .wav files, some that is associated with podcasts and certain video formats, etc., and the central identity of the iTunes database that may lead to loss of information when any Apple devices are subsequently synced with the rebuilt library.

Internal IDs are used by the script to access each object under consideration. They are the primary key to the database if that term means anything. The script also uses several alternate keys. For example I've used the term logical duplicate to describe the situation where two different entries in the library actually connect to the same physical file. This works by creating an index of all of the paths of the tracks that have been looked at so far in the current pass. If the same path crops up twice there is a duplicate, and the script then works out the optimum way to remove it. Physical dupes have the same values for artist, album, name, track & disc number, and size. Alternate dupes have the same tag properties, but have different sizes. Missing dupes have the same properties, but one track can be accessed, while another cannot. Playlist dupes refer to the same object twice in the same playlist. Pseudo-physical dupes arise when a single file has been imported via two different paths because there are drive mappings or folder redirections in place.

Say a user simply moves their media folder to a new drive by hand rather than letting iTunes manage the task, and then reimports the media to the library. Now each track is duplicated, one copy that plays, and another that doesn't, but all the stats and playlists are connected to the broken tracks. The script does a neat job of tidying up. Trashing the ITL and importing will leave a deduplicated library, but all the stats and plalists have been lost too.

tt2

I tested it. The playlist duplicates cannot be fixed without the library remaining intact. For some reason by re-importing the playlist after the file is removed and a new library is built, the library still thinks there are two physical files in the playlist even though the second no longer exists. The library was able to generate a second database entry based on the playlist text file information alone! This also tells me playlists are stored in a separate table, and or have their own location, which I think you can confirm. One other option would be to just walk through the playlists and remove the second entries by getting the end of line for the song name.

#1

Existence August Burns Red August Burns Red Constellations Metal 6854891 233 2 12 2009 1/19/2011 6:34 PM 11/25/2015 12:16 PM 234 44100 MPEG audio file “Constellations” is heavier, more technical, faster and more diverse than any August Burns Red album to come out yet. I was managing my expectations when August Burns Red announced “Constellations” because I loved “Messengers”, I didn’t know if they could E:\Music\iTunes\iTunes Media\Music\August Burns Red\Constellations\02 Existence 2.mp3

#2

Existence August Burns Red August Burns Red Constellations Metal 6854891 233 2 12 2009 1/19/2011 6:34 PM 11/25/2015 12:16 PM 234 44100 MPEG audio file “Constellations” is heavier, more technical, faster and more diverse than any August Burns Red album to come out yet. I was managing my expectations when August Burns Red announced “Constellations” because I loved “Messengers”, I didn’t know if they could E:\Music\iTunes\iTunes Media\Music\August Burns Red\Constellations\02 Existence.mp3

Most of the pertinent information about the song remains. I never had ratings, but I can see how that would also be something some users would want to keep if they are in fact lost. I didn't test play counts or other info like that either, it could be an issue as you say.

I just started a new thread referencing this information, but...

Could you please talk a little more about making iTunes look to the external hard drive for the iTunes (media folder, if that's what it's called)?

I have all my music on an external hard drive (except for some new purchases I just noticed are saved in the "cloud" and not on my ext. hard drive; a question for later perhaps - how do I get the files of my purchases on my ext. hard drive instead of on the "cloud" which I really don't use/take advantage of). I also have the annoying problem of double and triple duplicates at times, related it seems to the letter of the "port" my ext. hard drive is connected to changing. I did the "hold down shift as iTunes starts up to create an iTunes (media folder?) on my ext. hard drive." Now what do I do? Copy and paste all my music into that? What part of it? At that point iTunes will always look there for my music? Should I keep my original music folder on the ext. hard drive as a back up?

My inclination is to clear out iTunes and start over with a clean slate. I've done that before with the configuration I have, but it sounds like the "holding the shift key as iTunes starts up" is they way to go to keep the duplication from happening. I read all the posts, and I don't really understand anything about script debugging and so forth.

It seemed logical to keep my rather large music folder on an external hard drive so as not to have too much stored on my computer. It seems there are mixed opinions on this method.

I'm a novice at all this, but trying to understand. Thank you for any help you can offer.

I just started a new thread referencing this information, but...

Could you please talk a little more about making iTunes look to the external hard drive for the iTunes (media folder, if that's what it's called)?

I have all my music on an external hard drive (except for some new purchases I just noticed are saved in the "cloud" and not on my ext. hard drive; a question for later perhaps - how do I get the files of my purchases on my ext. hard drive instead of on the "cloud" which I really don't use/take advantage of).

I also have the annoying problem of double and triple duplicates at times, related it seems to the letter of the "port" my ext. hard drive is connected to changing. I did the "hold down shift as iTunes starts up to create an iTunes (media folder?) on my ext. hard drive."

Now what do I do? Copy and paste all my music into that? What part of it? At that point iTunes will always look there for my music? I see that (under Edit, Advanced Preferences) iTunes is looking at "F" which is currently the port my ext. hard drive is plugged into. If that letter changes, as it sometimes does, iTunes will still know where to look? Should I keep my original music folder on the ext. hard drive as a back up?

My inclination is to clear out iTunes and start over with a clean slate. I've done that before with the configuration I have, but it sounds like the "holding the shift key as iTunes starts up" is they way to go to keep the duplication from happening. I read all the posts, and I don't really understand anything about script debugging and so forth.

It seemed logical to keep my rather large music folder on an external hard drive so as not to have too much stored on my computer. It seems there are mixed opinions on this method.

I'm a novice at all this, but trying to understand. Thank you for any help you can offer.

Sorry to duplicate: the post from 9:21 has more thoughts; couldn't delete first post. Apologies for my lack of computer savvy.

See response in your other thread on Requesting more information: iTunes library on external hard drive.

I have had an iPhone and using iTunes for over 6 years now. I have dealt with this duplicate issue and tried to manage, often finding it complicated and avoiding the issue altogether leaving my music in total disarray.

Recently I have finally talked my husband into leaving the Droid world and crossing over to iPhone. Both of our teenage daughter also have iPhones now as well.

I have a iMac.

I am trying to compile years and years of music collections onto my iMac in a shared folder. I am creating multiple user accounts on my iMac so that every individual can plug in their phone and sync to their own iTunes account and manage their own apps, photos, music, etc on their own.

I have created the shared file, and I have created everyones user account. I want to manage the music correctly the first time. If this becomes a complicated issue, my husband will abandon the cross over to the iPhone, and my teenagers will surely only complicate media management even further, leaving a disaster for me to clean up.

I NEED simplicity! I am tech challenged person! I have been reading this thread and it is all greek to me. I do not understand anything being said.

One thing I love about Apple products is how simple they are to use, and user friendly. Unfortunately I do not find that to be true when dealing with iTunes and music management.

I do not want duplicates every time I open iTunes, I also do not want missing files every time I try to listen to something, or try to sync to my phone.

Could you please give me the simplest step by step instructions on how to manage this correctly from the very start? I am a overwhelmed military wife, mother of 2 busy teenagers, with more things being added to my to-do list than I can keep up with. I need to get this done in a day....or it will never be completed!

Starting with my Shared File in which I am moving all my music to so that each multiple user can access the content.......... what do I do next?

Step #1...........

It would be very nice if iTunes would handle the duplicate issue for me. I have a 7TB media server with all of the families music on it and I've made a symlink from /iTunes/iTunes Media/Automatically Add to iTunes/ to the server. Every time I start iTunes, it will add of the songs again (and again...) and I have since deleted the symlink. If there was a way to have iTunes not add duplicates, this would be awesome! By duplicate, I mean same everything; title, album, artist, length, etc.

Scott

I suspect that the symlink is the problem. iTunes will only duplicate library entries when:

• you add media files that are already present in the library, and
• those media files are not within the iTunes Media folder structure

The behavior you describe suggests that files are being added to the Automatically Added to iTunes folder over and over again, or that iTunes is unable to delete files from this folder after it has added them the first time (this could be an issue with WIndows permissions).

iTunes doesn't play nicely with symlinks and the media folder. You can redirect iOS backups with one, but that is a separate issue, and even then I would recommend a locally connected drive, not a network share.

tt2