Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

You had asked what I meant by encrypted hidden files, that is what iTunes uses to populate it's database, and it is the reason all of it's advanced features under preferences are redundant. There was a time when your ipod was potentially the only remaining stored location of your music and you had to go this route to rebuild your library. I don't know why you keep saying use deduper, that or another program would work the same...it's up to the user:

1. Backup your music to an external drive or another location on your physical computer drives.

2. If you already have double files, use a Duplicate file remover such as tt2's script DeDuper or another program such as Digital Volcano's Duplicate File Remover Pro. iTunes could, but never will make this feature.

3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.

4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.

5. Close iTunes

6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes

7. Open iTunes and wait for everything to populate.

8. Change any settings you would like to use under preferences > advanced from then on, but follow steps 1 to 6 for new stuff.

Why would anybody want to use steps one though six every time they add some content to the library? Hopefully this isn't a situation anyone gets into more than once. Moreover most people will want to preserve as much of the effort previously invested in building their library as possible when deduping which is why I want to highlight the particular drawback that your suggestion to use a file based method and then rebuild the library from scratch results in. If sombody uses the DeDuper script at step 2, then 3 through 8 are redundant.

This comment:

don't have it do anything else because none of the other features work.

suggests that both hhgttg27's and my efforts to explain how those features work and exactly what their impact is have failed.

tt2

' Create a string representing the 64 bit persistent ID of an iTunes object

' Modified 2012-08-24

Function PersistentID(T)

PersistentID=Right("0000000" & Hex(iTunes.ITObjectPersistentIDHigh(T)),8) & "-" & Right("0000000" & Hex(iTunes.ITObjectPersistentIDLow(T)),8)

End Function

Also, the script is just replacing/ removing the entries in the itl, can't you just delete the itl and generate a new one with the steps so that you don't have all that extra code? Did you write this tt2? If you did I would think you know about the encoding of the database string from the script. The playlists are separate files are they not? You can export them and re-import them? Let me know what you think.

I recognize my own code thanks. The script is just a program like any other. You run it, it does what it is supposed to do. Nobody needs to look inside it to make use of it. I thought you were actually interested in what is going on under the hood both with iTunes and the script. No problem if you aren't.

tt2

Your code does works great. iTunes also rebuilds the library for you, so you could just allow it to run and regenerate the library from all of the other files without needing to call the ID's from the database and count through the appendices to get the duplicates. It should just update the playlists since they are looking for links that won't exist anymore. A simple test would be to create a playlist with duplicate files, close iTunes, delete the itl., delete the physical file except for one, and then reopen iTunes. If it works you know you can delete it without fault. The information about the music file itself won't get lost because once iTunes opens it, either the info gets pulled from the file or from the web, so artwork and info isn't an issue either.

Now, you may be using the iTunes ID's to find duplicates, in which case if you're in the .itl already and you are already counting through the IDs then you might as well use what you are storing in those classes. If the script is doing that, then that makes more sense to me. That also means those aren't just a random collection of bits, they are your key to finding duplicates. If you are looking in the physical directories for dupes, then I think the simple way would be to allow iTunes to regenerate the itl, and playlist files.

I've tried to stress that as well as the standard tag data that is imported when tracks are added to a new empty library there is plenty of data that isn't; ratings, playlists, play & skip counts & dates, all metadata associated with .wav files, some that is associated with podcasts and certain video formats, etc., and the central identity of the iTunes database that may lead to loss of information when any Apple devices are subsequently synced with the rebuilt library.

Internal IDs are used by the script to access each object under consideration. They are the primary key to the database if that term means anything. The script also uses several alternate keys. For example I've used the term logical duplicate to describe the situation where two different entries in the library actually connect to the same physical file. This works by creating an index of all of the paths of the tracks that have been looked at so far in the current pass. If the same path crops up twice there is a duplicate, and the script then works out the optimum way to remove it. Physical dupes have the same values for artist, album, name, track & disc number, and size. Alternate dupes have the same tag properties, but have different sizes. Missing dupes have the same properties, but one track can be accessed, while another cannot. Playlist dupes refer to the same object twice in the same playlist. Pseudo-physical dupes arise when a single file has been imported via two different paths because there are drive mappings or folder redirections in place.

Say a user simply moves their media folder to a new drive by hand rather than letting iTunes manage the task, and then reimports the media to the library. Now each track is duplicated, one copy that plays, and another that doesn't, but all the stats and playlists are connected to the broken tracks. The script does a neat job of tidying up. Trashing the ITL and importing will leave a deduplicated library, but all the stats and plalists have been lost too.

tt2

I tested it. The playlist duplicates cannot be fixed without the library remaining intact. For some reason by re-importing the playlist after the file is removed and a new library is built, the library still thinks there are two physical files in the playlist even though the second no longer exists. The library was able to generate a second database entry based on the playlist text file information alone! This also tells me playlists are stored in a separate table, and or have their own location, which I think you can confirm. One other option would be to just walk through the playlists and remove the second entries by getting the end of line for the song name.

#1

Existence August Burns Red August Burns Red Constellations Metal 6854891 233 2 12 2009 1/19/2011 6:34 PM 11/25/2015 12:16 PM 234 44100 MPEG audio file “Constellations” is heavier, more technical, faster and more diverse than any August Burns Red album to come out yet. I was managing my expectations when August Burns Red announced “Constellations” because I loved “Messengers”, I didn’t know if they could E:\Music\iTunes\iTunes Media\Music\August Burns Red\Constellations\02 Existence 2.mp3

#2

Existence August Burns Red August Burns Red Constellations Metal 6854891 233 2 12 2009 1/19/2011 6:34 PM 11/25/2015 12:16 PM 234 44100 MPEG audio file “Constellations” is heavier, more technical, faster and more diverse than any August Burns Red album to come out yet. I was managing my expectations when August Burns Red announced “Constellations” because I loved “Messengers”, I didn’t know if they could E:\Music\iTunes\iTunes Media\Music\August Burns Red\Constellations\02 Existence.mp3

Most of the pertinent information about the song remains. I never had ratings, but I can see how that would also be something some users would want to keep if they are in fact lost. I didn't test play counts or other info like that either, it could be an issue as you say.

I just started a new thread referencing this information, but...

Could you please talk a little more about making iTunes look to the external hard drive for the iTunes (media folder, if that's what it's called)?

I have all my music on an external hard drive (except for some new purchases I just noticed are saved in the "cloud" and not on my ext. hard drive; a question for later perhaps - how do I get the files of my purchases on my ext. hard drive instead of on the "cloud" which I really don't use/take advantage of). I also have the annoying problem of double and triple duplicates at times, related it seems to the letter of the "port" my ext. hard drive is connected to changing. I did the "hold down shift as iTunes starts up to create an iTunes (media folder?) on my ext. hard drive." Now what do I do? Copy and paste all my music into that? What part of it? At that point iTunes will always look there for my music? Should I keep my original music folder on the ext. hard drive as a back up?

My inclination is to clear out iTunes and start over with a clean slate. I've done that before with the configuration I have, but it sounds like the "holding the shift key as iTunes starts up" is they way to go to keep the duplication from happening. I read all the posts, and I don't really understand anything about script debugging and so forth.

It seemed logical to keep my rather large music folder on an external hard drive so as not to have too much stored on my computer. It seems there are mixed opinions on this method.

I'm a novice at all this, but trying to understand. Thank you for any help you can offer.

I just started a new thread referencing this information, but...

Could you please talk a little more about making iTunes look to the external hard drive for the iTunes (media folder, if that's what it's called)?

I have all my music on an external hard drive (except for some new purchases I just noticed are saved in the "cloud" and not on my ext. hard drive; a question for later perhaps - how do I get the files of my purchases on my ext. hard drive instead of on the "cloud" which I really don't use/take advantage of).

I also have the annoying problem of double and triple duplicates at times, related it seems to the letter of the "port" my ext. hard drive is connected to changing. I did the "hold down shift as iTunes starts up to create an iTunes (media folder?) on my ext. hard drive."

Now what do I do? Copy and paste all my music into that? What part of it? At that point iTunes will always look there for my music? I see that (under Edit, Advanced Preferences) iTunes is looking at "F" which is currently the port my ext. hard drive is plugged into. If that letter changes, as it sometimes does, iTunes will still know where to look? Should I keep my original music folder on the ext. hard drive as a back up?

My inclination is to clear out iTunes and start over with a clean slate. I've done that before with the configuration I have, but it sounds like the "holding the shift key as iTunes starts up" is they way to go to keep the duplication from happening. I read all the posts, and I don't really understand anything about script debugging and so forth.

It seemed logical to keep my rather large music folder on an external hard drive so as not to have too much stored on my computer. It seems there are mixed opinions on this method.

I'm a novice at all this, but trying to understand. Thank you for any help you can offer.

I have had an iPhone and using iTunes for over 6 years now. I have dealt with this duplicate issue and tried to manage, often finding it complicated and avoiding the issue altogether leaving my music in total disarray.

Recently I have finally talked my husband into leaving the Droid world and crossing over to iPhone. Both of our teenage daughter also have iPhones now as well.

I have a iMac.

I am trying to compile years and years of music collections onto my iMac in a shared folder. I am creating multiple user accounts on my iMac so that every individual can plug in their phone and sync to their own iTunes account and manage their own apps, photos, music, etc on their own.

I have created the shared file, and I have created everyones user account. I want to manage the music correctly the first time. If this becomes a complicated issue, my husband will abandon the cross over to the iPhone, and my teenagers will surely only complicate media management even further, leaving a disaster for me to clean up.

I NEED simplicity! I am tech challenged person! I have been reading this thread and it is all greek to me. I do not understand anything being said.

One thing I love about Apple products is how simple they are to use, and user friendly. Unfortunately I do not find that to be true when dealing with iTunes and music management.

I do not want duplicates every time I open iTunes, I also do not want missing files every time I try to listen to something, or try to sync to my phone.

Could you please give me the simplest step by step instructions on how to manage this correctly from the very start? I am a overwhelmed military wife, mother of 2 busy teenagers, with more things being added to my to-do list than I can keep up with. I need to get this done in a day....or it will never be completed!

Starting with my Shared File in which I am moving all my music to so that each multiple user can access the content.......... what do I do next?

Step #1...........

It would be very nice if iTunes would handle the duplicate issue for me. I have a 7TB media server with all of the families music on it and I've made a symlink from /iTunes/iTunes Media/Automatically Add to iTunes/ to the server. Every time I start iTunes, it will add of the songs again (and again...) and I have since deleted the symlink. If there was a way to have iTunes not add duplicates, this would be awesome! By duplicate, I mean same everything; title, album, artist, length, etc.

Scott

iTunes doesn't play nicely with symlinks and the media folder. You can redirect iOS backups with one, but that is a separate issue, and even then I would recommend a locally connected drive, not a network share.

tt2

That's why it would be nice if iTunes did not add exact duplicates!

I don't want to drag-n-drop my songs into a different folder (automatically add...), I just want to put them in one folder and not mess with it. It would make my life simpler.

Scott

This does not really help me. I recently bought a new computer (with Windows 10) and downloaded iTunes. I transferred my music to my new PC, but when I upload folders to iTunes, it duplicates my files so I get 2, 3, and even 4 copies of songs. I have tried File --> Add Folder and I have tried drag and drop. I have tried doing it in small increments...just a few letters of the alphabet at a time... and it will go well until I get about halfway through the alphabet, then duplicates everything already there. None of them have the exclamation mark and my music library is too big to delete the songs one at a time. I just spent an entire day on this, only to have to delete my whole iTunes library (again). How do I get this to work correctly and prevent this problem from recurring?

See response in your other thread on Requesting more information: iTunes library on external hard drive.