Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

Like any other piece of software it works the way its programmers have told it to. That's not always exactly what they intended to tell it to do and, more often than not, not exactly the way you would like or expect it to.

Things to bear in mind:

1. If you have iTunes set up to Copy files to iTunes Media folder when adding to library (default option) and then repeatedly import a folder that is outside the iTunes Media folder into the library, it will do as it is told and create a new set of copies of all those files.
2. If you add a folder to iTunes which already has a bunch of duplicate files in it, iTunes will add in all the duplicates
3. If you add in untagged files then iTunes will lump them all into Unknown Artist/Unknown Album and set the track name to <Filename>.

Solutions:

1. Change your workflow. Move new stuff inside the iTunes Media folder before adding it to iTunes or disable the Copy to... option.
2. Manually dedupe before importing or see my DeDuper script for use after the fact.
3. Disable the Copy to... option and use my TagFromFilename script or fix the tags before importing into iTunes. (MediaMonkey's AutoTag From Filename is quite good for this).

tt2

iTunes doesn't monitor your media folders in the way that Windows Media Player does. iTunes only offers to "collect" files on the first launch after the initial install. Depending on the options involved that can create duplicates. For example if you have albums already imported into Windows Media Player and sitting in the default <User's Music> folders as <User's Music>\<Album Artist>\<Album> then iTunes can create the first set of dupes (although they are unique to iTunes at this point) as <User's Music>\iTunes\iTunes Media\Music\<Album Artist>\<Album>.

With your media held on an external drive you would probably be better off placing your iTunes Library there too. If you hold down shift as you start iTunes you can create a new empty library at X:\iTunes where X: is your external drive letter. If you already have a device attached to the current library, or ratings, playlists etc. that you want to preserve then it might be better to copy your existing iTunes folder out to X:\iTunes and then use the shift-start iTunes method to connect to it.

If you start fresh then just move everything into X:\iTunes\iTunes Media\Music before adding it to iTunes (up to you whether or not you want iTunes in charge or organising the structure i.e. "Keep iTunes Media folder organised").

If you want to sort out existing content into your relocated library use my ConsolidateByMoving script to, um, consolidate by moving files instead of copying them.

For tips on getting WMP and iTunes to play nice see this thread.

tt2

They do "just work", until you do something unexpected with them.

See this post for more info. on the library folder. The crucial file is iTunes Library.itl which is the central database of what is in your library.

I've assumed your netbook's drive is too small to hold your library so you would want to keep the media on an external drive. Having a split library where the database & library files are on the internal drive while the media is on another drive makes problems more likely. E.g external drive letter changes because you connect another device. If I'm wrong then just go for it and defupe when you are done. I'd still recommend you follow the advice on WMP's options to make sure it doesn't go moving files from inside the iTunes Media folder.

tt2

Thank you for such a thorough and helpful reply - greatly appreciated.

Thanks so much for this wonderful little utility! You've saved me hours of work manually deleting duplicates. Much appreciated!

You're welcome. 🙂

tt2

First thing first. This thread is currently at the top of the google search list for duplicate files in iTunes. So I feel a necro-post may not be entirely inappropriate.

I am coming over from Zune. I loved the software player for my zune but alas the hardware and software are dead so I must move on. I am not here to praise the Zune and condemn iTunes but rather to try and learn what it is that iTunes expects of me for a harmonious relationship(because it is not obvious from what I have been using). Rather than rehashing it all I feel I can leverage some of the advice already given and push this thread along on topic. I should state that I am however fundamentally opposed to any software that requires 3rd party fixit scripts to accomplish basic operations that should have been possible through the original software. So while you have likely done a superb job of helping iTunes along I am not going to run any 3rd party scripts.

(1) I have a large library of media that resides on a network drive for the convenience of sharing throughout the users on the network as well as for simple portability. I noticed right away that adding this music was not at all simple or obvious. But with the help of some tricks I was able to turn on the toolbar in itunes. And from there I pointed itunes at my network directory. Because this seemed like abnormal use of the software I wonder if this is not the preferred way to add music to itunes. From the posts above it seems iTunes would have preferred I move all my music to the local C:\Users\NICK\Music\iTunes\iTunes Media\Music directory. Is that correct? Is that how you would ideally continue to add new music into itunes as well?

(2) Assuming my assertion in 1 was correct it must not be possible to use a network or removable drive as your itunes library right? That seems very backwards from how I see people manage their music. But then again I am a guy with thousands of duplicates now, so what do I know.

(3) When I added my library there were a fairly sizable subset of files that were in a non-compatible file type. I let itunes convert these and I am pretty sure it placed the conversions into the local directory rather than placing them with the rest of my library. I confirmed this to be true and it did not cause a duplicate issue at that time.

(4) Thinking that iTunes would actively scan the library I pointed it at originally, I proceeded to add in a few dozen new albums to my network library that I had on another drive. I was surprised to see that they did not auto detect and come right in like I was used to with Zune player. No problem I just re pointed at the network drive and in they came... along with duplicates of all the previously added files. This behavior is pretty hard for me to justify in my mind. So Itunes is clearly pointing at all those network files for playback but it is unable to notice new additions in the same directory and a refresh causes it amnesia to the fact that it already added the files it is still pointing at? This must be the RPN of file management. Can someone please explain to me how this was meant to be used? The best I can guess is that iTunes frowns on external storage and has intentionally created this abusive flow to push us away from external storage?

(5) Something interesting was said above about changing the itunes\media\music directory to also reside on the external drive. If I were to wipe the slate clean and change my itunes directory to be the same as my network media drive would this allow itunes to scan for changes automatically and eliminate the need to add folders manually? I am concerned that I would still end up with the converted files being added to my originals as duplicates of a different file type. But I suppose I could live with that.

(6) Finally, I am considering iTunes match subscription exclusively for the high bit rate replacement of music already in my library. But first I want to know what apple plans to do. Will iTunes place those new higher bit rate files in the itunes\music directory among what is already there or will it separate the new from the old so I can easily eliminate the lower bit rate old files afterward?

Thanks in advance for any thoughts or answers

-Nick-

If you've used Windows Media Player in the past you will be familiar with the way that it behaves. You can specify one or more watched folders and it will integrate anything in those folders into the library automatically. I imagine the Zune player was somewhat similiar. iTunes is different. With iTunes you specify a single media folder. It doesn't automatically rescan that folder, although there is an additional subfolder that it creates called Automatically Add to iTunes. Anything you put in there will be added to iTunes and moved to where iTunes thinks it belongs, typically <Media Folder>\Music\<Artist>\<Album>\## <Name>.<Ext>.

There are three options that control the behaviour when new media is added to the library, located under Edit > Preferences > Advanced. (Turning on the initial hidden menu bar with Ctrl+B may be needed here.)

• The media folder location is where iTunes will place new content that is ripped, converted or downloaded with iTunes, or any items added to the library when the Copy file... option is enabled. This folder is normally located at <User's Music>\iTunes\iTunes Media. It is possible to change the media folder, and also the location of the library files.
• Keep iTunes Media folder organized lets iTunes rearrange files inside the media folder when you make tag changes.
• Copy files to iTunes Media folder when adding to library tells it to make copies inside the iTunes media folder of any files that are located outside of the iTunes Media folder when you add them to iTunes.

The last two options are enabled when iTunes is first installed.

iTunes knows the locations of all the files that are currently connected to the library. If you use File > Add Folder to Library, or a drag & drop action to, for example, add the current media folder, then only new files that you have manually placed in that folder will be added. If, on the other hand, you have the Copy files... option enabled, and rescan a folder located outside of the media folder then you are telling iTunes to import a new set of duplicate copies of those files. It won't detect that you're making duplicates. Import that folder again and you make more duplicates each time.

If you want to use a shared network folder as the source of media for multiple libraries then turn off the Copy files... option so that each file can be referenced on its original path without any copies being made of it. You can also tell iTunes to use the network folder as its media folder, but you should probably turn off the Keep... option so that an edit in one library doesn't move a file and break the links for the other.

1. iTunes doesn't mind where you decide to keep your music, however ripped tracks, purchases and downloads will always go into the designated media folder in the standard layout. Where things live shouldn't generally matter as long at they can be accessed when needed. Since this is usually via an application rather than a file browser my advice is normally to let iTunes get on with it. With a shared media folder it is better to take some manually control, but accept the overall structure that iTunes uses.
   
2. You can declare a shared network location (or indeed a folder on an external drive) as your media folder location. One potential drawback with this approach is that the media will be reset if iTunes starts and it is not able to access the remote path. If you don't realise at the time then you can end up with files in both locations and some remedial work to consolidate everything back to the main media folder and clean the unwanted copies in the local machine.
   
3. That sounds right. iTunes won't play WMA files natively, but can make copies in your chosen import format. However it hasn't kept a record of which files have been converted so if you add the source folder again it would convert those same files, again making duplicates.
   
4. Unless I've missed something iTunes isn't pointing to the network location, it has made its own copies of the inital set of imported files in the iTunes Media folder. Asking it to scan that folder again has made a new set of copies, importing the additions, but duplicating what had previously been imported.
   
5. Ideally you would change the media folder to point to something like \\<Server>\iTunes\iTunes Media. You would then move your media inside the iTunes Media folder before adding it to an empty iTunes library. This suggested path would allow you to move the library files into the otherwise empty iTunes folder at some future point if/when you want to move the library to a new drive or machine. Having the library files in the "correct" relationship to the media folder makes the library portable. Moving the library when it isn't portable can result in the library not being able to locate the media after it has been moved. iTunes will make subfolders for various media types within the media folder, e.g. Music, Movies, Podcasts, etc. so whatever path you choose it shouldn't end with Music.
   
6. I don't use iTunes Match, but my understanding is that it doesn't automatically replace your existing content. I think you have to manually delete your originals and then download their matched counterparts. However it works you probably ought to plan your own archive of your originals before you start. Potential pitfalls include having explit originals replaced by clean, albeit better quality, versions.

Further reading:

Backing up: Backup your iTunes for Windows library with SyncToy

The genesis of the Deduper script: Re: how to delete duplicated songs all at one time?

Background on iTunes library structure: Make a split library portable

Using iTunes alongside other players: Getting iTunes & Windows Media Player to play nicely

Third party tool for adding folder scanning behaviour to iTunes: iTunes Folder Watch

tt2

Hello Turingtest2,

Thank you for the detailed response.

"iTunes is different. With iTunes you specify a single media folder. It doesn't automatically rescan that folder, although there is an additional subfolder that it creates called Automatically Add to iTunes. Anything you put in there will be added to iTunes and moved to where iTunes thinks it belongs, typically <Media Folder>\Music\<Artist>\<Album>\## <Name>.<Ext>."

I have one media library except for the other media sub-library that itunes insists on creating and oh yeah if I want to add new songs I can't just put them in my media library or my itunes library but rather a third location where itunes proceeds to decide where it goes from there? I hope I interpreted that incorrectly because it sounds like pure foolishness in my opinion and a terrible situation for file organization/backup.

"There are three options that control the behaviour when new media is added to the library, "

Ok. I have not ripped a disc in a while but good to know. I do have "Keep Organized" turned on but I wouldn't dare enable "Copy files" based on what I have seen thus far.

"iTunes knows the locations of all the files that are currently connected to the library. If you use File > Add Folder to Library, or a drag & drop action to, for example, add the current media folder, then only new files that you have manually placed in that folder will be added. If, on the other hand, you have the Copy files... option enabled, and rescan a folder located outside of the media folder then you are telling iTunes to import a new set of duplicate copies of those files. It won't detect that you're making duplicates. Import that folder again and you make more duplicates each time."

What I am trying to tell itunes is to synchronize not copy, but I will try to understand why itunes is not familiar with the synchronization concept.

"If you want to use a shared network folder as the source of media for multiple libraries then turn off the Copy files... option so that each file can be referenced on its original path without any copies being made of it. You can also tell iTunes to use the network folder as its media folder, but you should probably turn off the Keep... option so that an edit in one library doesn't move a file and break the links for the other."

So disappointing. This is not a working solution in my opinion. The whole idea of a shared drive is the collaborative input of all users. But this really seems to be the fundamental flaw I must accept. itunes will not scan for changes, so I must account for a lifetime of handholding in the event that anything would change(which it will on a regular basis).

1. For me the big hurdle is not having positional control over my libraries. That is how I keep everything organized across multiple platforms and regulate a strict backup regimen. If I consent that itunes is allowed to put files in other locations then I must hunt them all down and manually transfer them to the master library or there will be no master library and my backups are incomplete.

2. Delightful. But like you already brought up it still requires manual addition for anything new. Duplicate cleanup is no fun either.

3. And it did. After deleting out all of the duplicates files however itunes persisted that two copies existed for all converted files. Also interesting is that iMatch also had duplicate listings from the same physical file. That is to say when removing duplicates I had to stipulate that it leave the actual files untouched or the one and only remaining file would be deleted from the local library and there would be no reference for the remaining original(other than icloud). This is different than I expected from your reading, but you did say you have not done iMatch.

4. No actually. I have verified that only a subset of my added files were copied over to itunes controlled directory when I pointed at the network library. I am fairly confident that only the files it converted from WMA ended up in that directory on the bulk add. And those are the same files that I ended up having duplicate issues with. I have a suspicion that I could push the converted files back into my original library, remove everything from itunes and re-point at the network drive to add without copying anything to the itunes media directory. But I have already wasted so much time on this transition I am not sure I have the desire to try yet... and I don't want all the duplicates in my master directory. I also don't want to trade my originals for a conversion of the original in yet another proprietary format.

5. Yes of course. Because my library is organized and on the network backups are in place for such risky occasions. You are correct about the replacement as well. For the low bitrate albums that I want to replace I simply delete the local file and download the iMatch version. I did encounter one very bizzare thing though that has me convinced I should not do this anymore. I was listening to a track on my zune while mowing the lawn and came inside to play it again from imatch cloud on my ipad... the track was different. I spent some time verifying that they should be the same. Same album, same track number, same length even(which was not common at almost 14 minutes long) but the content was different. So I am concerned that my dreams for a synchronized library through iMatch may be more like iResemble.

Thank you again for the time you put into answering my questions. I feel I have the information I need now so no further responses are requested by me at this time.

-Nick-

I, too, am having an 'Unknown Album' crisis, but slightly different to yours.

I signed up to iTunes Match earlier in the year and I noticed today that it had matched music that was in my library with files names like Track 1, Track 2.
To find out what tracks they were, I was able to use the 'Remove Download' function, download the song back into the library, then use the 'Go to iTunes Store' to find out what the music was. The annoying thing (as far as I can tell) is that even though iTunes knows what the track is, there's no way to update the info automatically. In an attempt to give it a nudge, I basically checked every preference that required some sort of communication with the iTunes store - I think it was just Automatically Download Album Artwork and the one about sharing library information. Match already had that so what could be the harm?

So....iTunes can now not locate a significant portion of my library. Luckily I had exported (parts of) my library data and I figured out that it was only files that were in an 'Unknown Album' folder of their respective artists, but not every 'Unknown Album' folder was affected. This is what I think happened, based on changes to file locations and sizes.

• iTunes has gone searching for album artwork and found what it thinks is a match.
• If I don't have any album info on a match that been found, for whatever reason iTunes is completing that based on the artwork.
• iTunes is then automatically updating it's file structure to reflect the changes made but, for some reason, it's not updating the location information.
• iTunes then can't find the file.

I started manually relocating the files, but that was excruciating. What I've been doing instead is deleting the file it can't find via 'Remove Download' and then downloading the tracks again. The downloaded tracks are about a MB larger than the original files, which I'm assuming is because they now come with artwork! Seems to be working so far...

The new problem I have is that I now have duplicates of a lot of music on my hard drive but they're not in my library. If anyone from Apple is listening, what's the best way to put the duplicates back into the library so I can remove them without having to trawl through each of the folders? Also, some of the artwork matches are waaaaaaaaay off - so it basically thinks that one song is something completely different. I'm worried this is going to affect my uploaded music in the cloud.

But in short, I don't have an answer to your problem, but I think your hypothesis that iTunes is this problematic and non-functional is correct. Good luck with your issue.

I have iTunes 12.2 and have relocated the iTunes Media folder location. After doing this I "Organized Library" and "Consolidated Files". About 80% of the way through the operation stopped. I subsequently found the file that wouldn't load and removed it. If I hit Consolidate Files again without changing the iTunes Media folder again to try to pick up the remaining 20% that didn't relocate will it duplicate files already relocated?

No, it should be fine. When files are consolidated iTunes checks the current location of each track. It only makes copies of the files located outside of the media folder, then updates the path for that entry to point to the new file.

tt2

Itunes duplicates playlist items. I'm using 12.2.2.25 at the moment and copying my music to a new computer. I can import a folder with single copies of songs. I export the playlists from the old computer. I then import my playlist into my new computer. Itunes makes a duplicate copy of the songs on my computer that are in the playlist. If the song is used in multiple playlists, the song is duplicated many times. If you use itunes duplicate removal method (view, show duplicates, select all, delete), the imported playlists are broken.

This is a real pain for those of us who use complicated play lists when we upgrade computers.