Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

I suspect that the symlink is the problem. iTunes will only duplicate library entries when:

• you add media files that are already present in the library, and
• those media files are not within the iTunes Media folder structure

The behavior you describe suggests that files are being added to the Automatically Added to iTunes folder over and over again, or that iTunes is unable to delete files from this folder after it has added them the first time (this could be an issue with WIndows permissions).

First thing first. This thread is currently at the top of the google search list for duplicate files in iTunes. So I feel a necro-post may not be entirely inappropriate.

I am coming over from Zune. I loved the software player for my zune but alas the hardware and software are dead so I must move on. I am not here to praise the Zune and condemn iTunes but rather to try and learn what it is that iTunes expects of me for a harmonious relationship(because it is not obvious from what I have been using). Rather than rehashing it all I feel I can leverage some of the advice already given and push this thread along on topic. I should state that I am however fundamentally opposed to any software that requires 3rd party fixit scripts to accomplish basic operations that should have been possible through the original software. So while you have likely done a superb job of helping iTunes along I am not going to run any 3rd party scripts.

(1) I have a large library of media that resides on a network drive for the convenience of sharing throughout the users on the network as well as for simple portability. I noticed right away that adding this music was not at all simple or obvious. But with the help of some tricks I was able to turn on the toolbar in itunes. And from there I pointed itunes at my network directory. Because this seemed like abnormal use of the software I wonder if this is not the preferred way to add music to itunes. From the posts above it seems iTunes would have preferred I move all my music to the local C:\Users\NICK\Music\iTunes\iTunes Media\Music directory. Is that correct? Is that how you would ideally continue to add new music into itunes as well?

(2) Assuming my assertion in 1 was correct it must not be possible to use a network or removable drive as your itunes library right? That seems very backwards from how I see people manage their music. But then again I am a guy with thousands of duplicates now, so what do I know.

(3) When I added my library there were a fairly sizable subset of files that were in a non-compatible file type. I let itunes convert these and I am pretty sure it placed the conversions into the local directory rather than placing them with the rest of my library. I confirmed this to be true and it did not cause a duplicate issue at that time.

(4) Thinking that iTunes would actively scan the library I pointed it at originally, I proceeded to add in a few dozen new albums to my network library that I had on another drive. I was surprised to see that they did not auto detect and come right in like I was used to with Zune player. No problem I just re pointed at the network drive and in they came... along with duplicates of all the previously added files. This behavior is pretty hard for me to justify in my mind. So Itunes is clearly pointing at all those network files for playback but it is unable to notice new additions in the same directory and a refresh causes it amnesia to the fact that it already added the files it is still pointing at? This must be the RPN of file management. Can someone please explain to me how this was meant to be used? The best I can guess is that iTunes frowns on external storage and has intentionally created this abusive flow to push us away from external storage?

(5) Something interesting was said above about changing the itunes\media\music directory to also reside on the external drive. If I were to wipe the slate clean and change my itunes directory to be the same as my network media drive would this allow itunes to scan for changes automatically and eliminate the need to add folders manually? I am concerned that I would still end up with the converted files being added to my originals as duplicates of a different file type. But I suppose I could live with that.

(6) Finally, I am considering iTunes match subscription exclusively for the high bit rate replacement of music already in my library. But first I want to know what apple plans to do. Will iTunes place those new higher bit rate files in the itunes\music directory among what is already there or will it separate the new from the old so I can easily eliminate the lower bit rate old files afterward?

Thanks in advance for any thoughts or answers

-Nick-

Hello Turingtest2,

Thank you for the detailed response.

"iTunes is different. With iTunes you specify a single media folder. It doesn't automatically rescan that folder, although there is an additional subfolder that it creates called Automatically Add to iTunes. Anything you put in there will be added to iTunes and moved to where iTunes thinks it belongs, typically <Media Folder>\Music\<Artist>\<Album>\## <Name>.<Ext>."

I have one media library except for the other media sub-library that itunes insists on creating and oh yeah if I want to add new songs I can't just put them in my media library or my itunes library but rather a third location where itunes proceeds to decide where it goes from there? I hope I interpreted that incorrectly because it sounds like pure foolishness in my opinion and a terrible situation for file organization/backup.

"There are three options that control the behaviour when new media is added to the library, "

Ok. I have not ripped a disc in a while but good to know. I do have "Keep Organized" turned on but I wouldn't dare enable "Copy files" based on what I have seen thus far.

"iTunes knows the locations of all the files that are currently connected to the library. If you use File > Add Folder to Library, or a drag & drop action to, for example, add the current media folder, then only new files that you have manually placed in that folder will be added. If, on the other hand, you have the Copy files... option enabled, and rescan a folder located outside of the media folder then you are telling iTunes to import a new set of duplicate copies of those files. It won't detect that you're making duplicates. Import that folder again and you make more duplicates each time."

What I am trying to tell itunes is to synchronize not copy, but I will try to understand why itunes is not familiar with the synchronization concept.

"If you want to use a shared network folder as the source of media for multiple libraries then turn off the Copy files... option so that each file can be referenced on its original path without any copies being made of it. You can also tell iTunes to use the network folder as its media folder, but you should probably turn off the Keep... option so that an edit in one library doesn't move a file and break the links for the other."

So disappointing. This is not a working solution in my opinion. The whole idea of a shared drive is the collaborative input of all users. But this really seems to be the fundamental flaw I must accept. itunes will not scan for changes, so I must account for a lifetime of handholding in the event that anything would change(which it will on a regular basis).

1. For me the big hurdle is not having positional control over my libraries. That is how I keep everything organized across multiple platforms and regulate a strict backup regimen. If I consent that itunes is allowed to put files in other locations then I must hunt them all down and manually transfer them to the master library or there will be no master library and my backups are incomplete.

2. Delightful. But like you already brought up it still requires manual addition for anything new. Duplicate cleanup is no fun either.

3. And it did. After deleting out all of the duplicates files however itunes persisted that two copies existed for all converted files. Also interesting is that iMatch also had duplicate listings from the same physical file. That is to say when removing duplicates I had to stipulate that it leave the actual files untouched or the one and only remaining file would be deleted from the local library and there would be no reference for the remaining original(other than icloud). This is different than I expected from your reading, but you did say you have not done iMatch.

4. No actually. I have verified that only a subset of my added files were copied over to itunes controlled directory when I pointed at the network library. I am fairly confident that only the files it converted from WMA ended up in that directory on the bulk add. And those are the same files that I ended up having duplicate issues with. I have a suspicion that I could push the converted files back into my original library, remove everything from itunes and re-point at the network drive to add without copying anything to the itunes media directory. But I have already wasted so much time on this transition I am not sure I have the desire to try yet... and I don't want all the duplicates in my master directory. I also don't want to trade my originals for a conversion of the original in yet another proprietary format.

5. Yes of course. Because my library is organized and on the network backups are in place for such risky occasions. You are correct about the replacement as well. For the low bitrate albums that I want to replace I simply delete the local file and download the iMatch version. I did encounter one very bizzare thing though that has me convinced I should not do this anymore. I was listening to a track on my zune while mowing the lawn and came inside to play it again from imatch cloud on my ipad... the track was different. I spent some time verifying that they should be the same. Same album, same track number, same length even(which was not common at almost 14 minutes long) but the content was different. So I am concerned that my dreams for a synchronized library through iMatch may be more like iResemble.

Thank you again for the time you put into answering my questions. I feel I have the information I need now so no further responses are requested by me at this time.

-Nick-

I have iTunes 12.2 and have relocated the iTunes Media folder location. After doing this I "Organized Library" and "Consolidated Files". About 80% of the way through the operation stopped. I subsequently found the file that wouldn't load and removed it. If I hit Consolidate Files again without changing the iTunes Media folder again to try to pick up the remaining 20% that didn't relocate will it duplicate files already relocated?

No, it should be fine. When files are consolidated iTunes checks the current location of each track. It only makes copies of the files located outside of the media folder, then updates the path for that entry to point to the new file.

tt2

Itunes duplicates playlist items. I'm using 12.2.2.25 at the moment and copying my music to a new computer. I can import a folder with single copies of songs. I export the playlists from the old computer. I then import my playlist into my new computer. Itunes makes a duplicate copy of the songs on my computer that are in the playlist. If the song is used in multiple playlists, the song is duplicated many times. If you use itunes duplicate removal method (view, show duplicates, select all, delete), the imported playlists are broken.

This is a real pain for those of us who use complicated play lists when we upgrade computers.

An external drive is the worst recommendation for a permanent location to be reading and writing music files. The amount of reads/writes listening to music would tax on an external drive is just ASKING for a drive failure. External drives are NOT designed for this unless you have purchased a very high end system configured for stability. I recommend no one attempt to read their large collections of music off of an external drive.

Additionally, I am going to second this issue and say I am confounded at the options and the results of the file storage for each setting.

"Keep folders organized" - created a music folder under my iTunes media folder and placed only a handful of artists splitting my music up and create duplicates there.

"Copy files" setting - Turned off when importing after moving everything to a music folder under iTunes Media folder I somehow still had duplicates so that feature isn't working.

"Changing iTunes media folder location" - You would think this is where the music should be, but the "iTunes Library" file is a directory above which created another set of duplicates: this is where I am seeing quadruple duplicate files. I'm going to see what deleting this file altogether does. Maybe it will trick itself into only seeing the music files once.

Placing files into "automatically added"- why would I put files into a folder that would cause the program to copy everything over and then run into the same issues of reading everything doubled up?

I'm assuming here that the hexadecimal encrypted folders that keep popping up when I try to import have something to do with it. If iTunes needs to read files that way, it is highly possible it cannot tell the difference between something it converts into it's special code and something that all other media players can normally just read. It's not like I can just switch either. I'm stuck with 3,000 purchased iTunes songs. A check and a stack of CD's would be nice at this point Apple....

Window shell shows all information is tied to the mp3, mp4, .flac, WAV whatever extension. -> iTunes places some music in at least 10 different folders named unknown artist. iTunes is clearly not designed to read music files other then their made up AAC extension

on a side and unrelated note iTunes tried to convert unsupported extensions to AAC. If I purchased music I don't need itunes to completely destroy the codec i purchased.

SIMPLE SOLUTION:

So, based on an amalgamation of the posts above I came up with a stupidly simple solution:

1. Backup your music to an external drive or another location on your physical computer drives.

2. If you already have double files, use a Duplicate file remover. iTunes could, but never will make this feature.

3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.

4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.

5. Close iTunes

6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes

7. Open iTunes and wait for everything to populate.

You should only see the exact files that you want in iTunes. If you still have duplicates you actually physically had duplicate files to begin with and you need to adjust how you remove them in the Duplicate file remover software such as "remove similar names" because "01 - Song" could be the same as "1-Song".

After your music is all loaded up you can delete everything out of the "Automatically Add to iTunes" folder. Unless you want to physically have 2 copies of the music on your drives.

The sad part about this solution is that in order to stop showing duplicate entries of music in iTunes you actually have to physically duplicate files. So, make sure you have enough space on your hard drive to temporarily store double the amount of data, otherwise it is impossible for you to use the iTunes program until you purchase a bigger hard drive.

The reason iTunes is showing the duplicate files is because if you didn't do it this way, and keep trying to adjust settings or add music directly to the folder that iTunes ends up creating for music, the library database keeps growing with duplicate entries. When iTunes goes to read what on the database it thinks there is supposed to be double of everything. It would make sense to just put the music in the folder it is supposed to go in, instead of a different folder that isn't very descriptive at all, but alas nothing makes sense in the Apple world.

Thanks for the settings definitions tt2. They reiterate your point earlier in the thread, and mine as well. Each setting does what it does, and you tell iTunes to either do them or not. However, there is no reason for someone to think through all of that on their to come up with the solution that I have already given above. Of course iTunes has an official solution, but that is useless when you think about the work of manually selecting everything. My approach is a third party duplicate remover that has a much more user friendly interface, but thank you for providing some source code. iTunes could begin supporting batch and control files that do all of this for one folder on windows, but like I said they won't.

You get what you pay for with external drives. The $60 range is going to fail 10 times more then the $130 range, but I would never buy something that big or expensive when I want portability. For portability and space your price skyrockets. https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/

NAS your talking even more money, which is also starting to get a little off topic here. The point is your suggestion is not advisable for a little Toshiba that someone uses to move large files, and have sit on a shelf the rest of the time and you have to really get into expensive and non portable alternatives to be reliable.

Thanks for the extra directory hierarchy httg. It is a good point that iTunes doesn't just duplicate, triple, or even quadruple songs, it does everything else too.

tt2 I've also considered your notes on the encrypted files and it seems there is more to them then artwork. If I were to copy songs directly off of an ipod, I wouldn't see names to these songs like a normal song file, I would get a hex based name under a set of hex named folders. Dropping them all into iTunes and playing them would decode the file and generate a copy in the iTunes media folder (assuming you copy these files off of an ipod to some other location). So, I am assuming that in order for the iTunes library.itl to talk with a number of apple devices, the database must store a hexadecimal data entry that can be decoded by each device. Since the HEX will be unique for each file it is impossible for iTunes to determine that they are the same or similar file, thus rendering the features of "Keep organized, consolidat, copy files etc." useless for all but one computer that has only iTunes bought songs on it. Additionally over time these hex entries in the database will just grow and grow and grow uniquely every time you copy stuff over or continue to use the settings when you get a new computer. You might have 2 physical copies of a song, or even one single copy, but once the Library.itl assigns a hex code to it, it has to generate a new one for it to consolidate, and do all that other stuff.

The point is there is only one way to do it quickly and effectively as I've listed above using a third party cleaner or running your source code, and never using the other settings again. Every single one of those settings and features has a potential to duplicate files, so none of them should be used because once it starts to multiply it's worse and worse until someone cuts it off at the root on a new build with a completely new library.itl.

"It is a good point that iTunes doesn't just duplicate, triple, or even quadruple songs, it does everything else too."

No, it doesn't - in nearly a decade of using iTunes and (once I'd decided to put my entire CD collection into it) build up a moderately large library (4500+ albums) I've never once seen a case where iTunes duplicated anything, except in cases where I'd mistakenly imported media file(s) that were already present in the library - and I doubt I've done that more than a handful of times.

Okay, I guess I missed what your point is with the illustrations. Looking more closely it seems you are saying that everything should be moved into the iTunes media folder, but that is what is causing duplicates for the users above. Also, what you are showing is for an apple product. Assuming you are using apple products exclusively all these years, I would say that you aren't going to run into these issues. Especially if you aren't manually moving files from one computer to another or one device to another like Zune (Windows product) to iTunes. Whenever you sync or plug in from one apple product to another it knows what to do because they all talk to each other off of the itl and xml files. Where the issue arises is a physical copy from one drive to another with something like an external drive.

This discussion is great and all, but there is a step by step solution to the original issue above.

iTunes stores the media files on an Apple device in an obfuscated fashion, using semi-random four character file names with the original file extension in a number of hidden folders. This is principally an anti-piracy measure. Apple have made it hard for somebody to load up their device with media, then take it around to a friend's and copy off selected albums. If you manage your library and its backup properly there is no need to delve inside the storage system on a device. They are not intended to be used to transport media from one library to another.

Duplicates in iTunes are always the result of the user making invalid assumptions about the consequences of some action which effectively instructs iTunes to make the copies. Using a Windows (or Mac) file deduper, then deleting the library database and reimporting works after a fashion, but throws away ratings, play counts, playlists, date added information, all metadata related to .wav files, etc. My script aims to avoid all of those drawbacks.

tt2

I don't know why that helps anyone here by saying that you haven't had issues. It sounds like you've followed all of the steps above when managing your music which is why you don't have duplicate files. The only thing I would say is you probably have duplicated music in your "automatically add to itunes" folder as well as your "music" folder under itunes media, which is a waste of space unless you physically have deleted those files yourself, which is part of my steps above....