Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

A few follow-ups to tt2's comments:

• I agree about the suitability of external drives - I've been using these for iTunes for the best part of a decade and not encountered any failure rates higher than internal drives. The one caveat that I would add is that "desktop" drives (larger form factor and with dedicated power supplies) are more reliable than "portable" drives that draw power over USB. As tt2 says all drives fall eventually, so a strategy for making and maintaining backups is essential.
• For those who like pictures 😁 this is the standard layout of iTunes media folders:
  
  I've yet to encounter a convincing argument for using a different structure or not allowing iTunes to manage where media is stored.
• Apple really should put something in the UI that states that the iTunes Media folder location setting in Edit > Preferences > Advanced does not change where iTunes looks for the existing content of your library.
• The interaction between the Copy files ... option in Edit > Preferences > Advanced and the different ways of adding media to the iTunes library can be summarized as follows:
  

If you've used Windows Media Player in the past you will be familiar with the way that it behaves. You can specify one or more watched folders and it will integrate anything in those folders into the library automatically. I imagine the Zune player was somewhat similiar. iTunes is different. With iTunes you specify a single media folder. It doesn't automatically rescan that folder, although there is an additional subfolder that it creates called Automatically Add to iTunes. Anything you put in there will be added to iTunes and moved to where iTunes thinks it belongs, typically <Media Folder>\Music\<Artist>\<Album>\## <Name>.<Ext>.

There are three options that control the behaviour when new media is added to the library, located under Edit > Preferences > Advanced. (Turning on the initial hidden menu bar with Ctrl+B may be needed here.)

• The media folder location is where iTunes will place new content that is ripped, converted or downloaded with iTunes, or any items added to the library when the Copy file... option is enabled. This folder is normally located at <User's Music>\iTunes\iTunes Media. It is possible to change the media folder, and also the location of the library files.
• Keep iTunes Media folder organized lets iTunes rearrange files inside the media folder when you make tag changes.
• Copy files to iTunes Media folder when adding to library tells it to make copies inside the iTunes media folder of any files that are located outside of the iTunes Media folder when you add them to iTunes.

The last two options are enabled when iTunes is first installed.

iTunes knows the locations of all the files that are currently connected to the library. If you use File > Add Folder to Library, or a drag & drop action to, for example, add the current media folder, then only new files that you have manually placed in that folder will be added. If, on the other hand, you have the Copy files... option enabled, and rescan a folder located outside of the media folder then you are telling iTunes to import a new set of duplicate copies of those files. It won't detect that you're making duplicates. Import that folder again and you make more duplicates each time.

If you want to use a shared network folder as the source of media for multiple libraries then turn off the Copy files... option so that each file can be referenced on its original path without any copies being made of it. You can also tell iTunes to use the network folder as its media folder, but you should probably turn off the Keep... option so that an edit in one library doesn't move a file and break the links for the other.

1. iTunes doesn't mind where you decide to keep your music, however ripped tracks, purchases and downloads will always go into the designated media folder in the standard layout. Where things live shouldn't generally matter as long at they can be accessed when needed. Since this is usually via an application rather than a file browser my advice is normally to let iTunes get on with it. With a shared media folder it is better to take some manually control, but accept the overall structure that iTunes uses.
   
2. You can declare a shared network location (or indeed a folder on an external drive) as your media folder location. One potential drawback with this approach is that the media will be reset if iTunes starts and it is not able to access the remote path. If you don't realise at the time then you can end up with files in both locations and some remedial work to consolidate everything back to the main media folder and clean the unwanted copies in the local machine.
   
3. That sounds right. iTunes won't play WMA files natively, but can make copies in your chosen import format. However it hasn't kept a record of which files have been converted so if you add the source folder again it would convert those same files, again making duplicates.
   
4. Unless I've missed something iTunes isn't pointing to the network location, it has made its own copies of the inital set of imported files in the iTunes Media folder. Asking it to scan that folder again has made a new set of copies, importing the additions, but duplicating what had previously been imported.
   
5. Ideally you would change the media folder to point to something like \\<Server>\iTunes\iTunes Media. You would then move your media inside the iTunes Media folder before adding it to an empty iTunes library. This suggested path would allow you to move the library files into the otherwise empty iTunes folder at some future point if/when you want to move the library to a new drive or machine. Having the library files in the "correct" relationship to the media folder makes the library portable. Moving the library when it isn't portable can result in the library not being able to locate the media after it has been moved. iTunes will make subfolders for various media types within the media folder, e.g. Music, Movies, Podcasts, etc. so whatever path you choose it shouldn't end with Music.
   
6. I don't use iTunes Match, but my understanding is that it doesn't automatically replace your existing content. I think you have to manually delete your originals and then download their matched counterparts. However it works you probably ought to plan your own archive of your originals before you start. Potential pitfalls include having explit originals replaced by clean, albeit better quality, versions.

Further reading:

Backing up: Backup your iTunes for Windows library with SyncToy

The genesis of the Deduper script: Re: how to delete duplicated songs all at one time?

Background on iTunes library structure: Make a split library portable

Using iTunes alongside other players: Getting iTunes & Windows Media Player to play nicely

Third party tool for adding folder scanning behaviour to iTunes: iTunes Folder Watch

tt2

A random collection of bits of information about iTunes. This is by no means everything that is germane to this topic.

• Internally an iTunes playlist is an ordered list of library persistent IDs. Changing metadata doesn't affect the IDs, though it can affect which tracks qualify for membership of smart playlists.
• An iTunes library can include multiple playlists, smart playlists and playlist folders. All of the lists and their structure is stored internally. Lists can be exported individually, but they don't ordinarily exist as separate files.
• An exported playlist is an ordered list of file paths. Changing metadata may affect where a file is stored, so an exported playlist cannot always be reimported without changes.
• Smart playlist rule sets are only stored in the .itl file. You can export a list, and reimport as a regular playlist, but you cannot export and then reimport the rules.
• When an Apple device is synced with the library, iTunes can compare the contents of the device in terms of persistent IDs with the selection of content in the library that is due to sync, and copy over only the new or changed media, and/or remove files that no longer qualify for the device. The IDs are also used to sync ratings, play and skip counts, and last played/skipped times.
• If you create a new library, even with the same media, each track will be assigned a new ID, breaking down the relationship between IDs on any device and those in the library.
• Each iTunes library has an internal ID which, among other things, iTunes uses to recognize when a device is connected to a new library. In such cases iTunes will normally want to remove existing content on the device and repopulate from the current library. This probably harks back in part to Apple's attempts to limit casual piracy but is also necessary because the IDs on the device won't match those in the library.
• Certain metadata including, but not limited to, data added, checked, rating, play and skip counts and times, and all metadata for .wav files is only stored in the .itl file. Creating a new library file causes all of this information to be lost.

Destroying the current .itl file and reimporting the media has unwanted side effects. Obviously some users may not care about the loss of the information, but for those that do it matters.

The script, which is indeed my work, makes a number of passes through the selected tracks identifying different types of duplicate and thins them out in a way that tries to preserve as much information as possible. For example if there are two copies of a song one of which has been added to playlist A, and another to playlist B, then after deduping the remaining copy of the song will be in both playlists. Play & skip counts are merged, with the remaining track getting the most recent play and/or skip time. Where a track exists in two different formats/bitrates the script will try to preserve the higher quality one, but the user can opt to favour the smaller, or one format over another. This also works even if there are 10s or 100s of playlists involved, and any number of duplicates of the same track. Where practical the script tries to preserve the earliest date added value. When duplicate files are deleted or archived, and the last media file is removed from a folder, then the script can also clean up any artwork or desktop.ini files that might have been generated by Windows Media Player and remove the empty folder. The script uses the same criteria as iTunes does for the Show Exact Duplicates feature. This allows different versions of the same song on different albums, or by different artists to be kept in the library. Working solely with filenames could easily get this wrong. The script is as complicated as it needs to be to work the way I think it needs to, and to accommodate options that others have asked me to include. I wrote it not because I have any need of it, but to see if I could. As I've noted in how to delete duplicated songs all at one time? deduping is not a trivial problem.

tt2

iTunes doesn't monitor your media folders in the way that Windows Media Player does. iTunes only offers to "collect" files on the first launch after the initial install. Depending on the options involved that can create duplicates. For example if you have albums already imported into Windows Media Player and sitting in the default <User's Music> folders as <User's Music>\<Album Artist>\<Album> then iTunes can create the first set of dupes (although they are unique to iTunes at this point) as <User's Music>\iTunes\iTunes Media\Music\<Album Artist>\<Album>.

With your media held on an external drive you would probably be better off placing your iTunes Library there too. If you hold down shift as you start iTunes you can create a new empty library at X:\iTunes where X: is your external drive letter. If you already have a device attached to the current library, or ratings, playlists etc. that you want to preserve then it might be better to copy your existing iTunes folder out to X:\iTunes and then use the shift-start iTunes method to connect to it.

If you start fresh then just move everything into X:\iTunes\iTunes Media\Music before adding it to iTunes (up to you whether or not you want iTunes in charge or organising the structure i.e. "Keep iTunes Media folder organised").

If you want to sort out existing content into your relocated library use my ConsolidateByMoving script to, um, consolidate by moving files instead of copying them.

For tips on getting WMP and iTunes to play nice see this thread.

tt2

They do "just work", until you do something unexpected with them.

See this post for more info. on the library folder. The crucial file is iTunes Library.itl which is the central database of what is in your library.

I've assumed your netbook's drive is too small to hold your library so you would want to keep the media on an external drive. Having a split library where the database & library files are on the internal drive while the media is on another drive makes problems more likely. E.g external drive letter changes because you connect another device. If I'm wrong then just go for it and defupe when you are done. I'd still recommend you follow the advice on WMP's options to make sure it doesn't go moving files from inside the iTunes Media folder.

tt2

You should clone the entire library, then there is no need to import anything, and no risk of creating duplicates.

See this migrate iTunes library post for suggestions on copying the library from your old computer to a new one.

tt2

Every external drive I've bought has got the same kind of 3.5" or 2.5" hard drive as is fitted internally to the desktop or laptop computers that I have bought. Internal is best if you have the room but a locally connected external works fine. A backup is vital. All drives will fail eventually, often without warning. The key factor is large. If the library is large enough not to fit on what is inside your box then it needs to go outside. A split library with media on NAS storage is a possibility, but iTunes doesn't play well with NAS and transfer rates are generally lower than with a local drive. RAID is great for those with the budget for it.

Let's try to explain the terms.

Keep organized controls what happens to files that are located inside the media folder when they are added to the library or if you make a change to their metadata after they have been added to the library. If the option is enabled the folder and file names may be edited to reflect the content of the tag, with music typically going to <Media Folder>\Music\<Album Artist>\## <Name>.<Ext> otherwise the file stays where it was. (There may be an optional leading disc number if the disc number or disc count is greater than 1, Artist is used if Album Artist is blank, and there is also an iTunes specific compilation flag that can put tracks in <Media Folder>\Music\Compilations<Album>.)

Copy files controls what happens to files that are outside the media folder when they are added to the library. If it is enabled iTunes makes a new copy inside the media folder based on the tag properties. It doesn't "remember" that the original has been imported so you are at risk of making duplicates if you import the same folder again. If the option is disabled iTunes connects to the file on its original path, the file's tag will be updated if you edit properties, but the file won't move.

Consolidate files tells iTunes to import files that are currently outside of the media folder in the same way that it would if they were added when Copy files was active. In and of itself consolidate files doesn't create duplicates within the iTunes library, but there is the potential for you to reimport the original files later unless these are actively archived or deleted after consolidating.

Anything ripped from CD, downloaded from the iTunes Store, converted to a new format, or added using the Automatically Add to iTunes folder is moved into the standard folder structure whether or not Keep organized or Copy files are enabled. A typical use case for Automatically Add to iTunes would be if you had a drive with media brought over from another computer that you want to add to iTunes. You can either add the folder when Copy files is active, or copy the folder of media into Automatically Add to iTunes. You shouldn't ever need to use the folder as part of moving an existing library from one computer to another. An advantage of using the folder will be that anything that iTunes cannot handle will be moved into a folder called Not Added. if you just add the folder in the normal way iTunes will silently ignore any track it doesn't like but you will be none the wiser.

Changing the media folder location tells iTunes where new media will be stored. When you make the change iTunes will typically offer to consolidate files to the new location, but you don't have to let it do so. If you want to change which library is opened you need to hold down shift (Windows) or atl/option (Mac) as you launch iTunes and then use the options to choose a library or create a new one.

Apple don't refer to the library folder as such, but it is the folder that contains the library database file, normally called iTunes Library.itl, along with some supporting files, the Album Artwork cache folder and, typically, the media folder iTunes Media. When all content connected to the library is inside the media folder, and the media folder is directly inside the library folder, then the library is in a portable shape that can be easily moved from one computer, drive, or path to another with out breaking the links between the library and the media. More at Make a split library portable.

Files in .wav format do not carry a tag. Windows Explorer shows no additional information for them. iTunes will import these to the Unknown Artist\Unknown Album folder. If they are still on paths that reflect Artist and Album after they are added to the library (which will depend on original location, media folder location and choices for Keep & Copy) then I have a script that will read in the data from the path: TagFromFilename. If cannot help if the files have been moved into Unknown Artist\Unknown Album unless there is an alternate undamaged source.

Occasionally .mp3 files have multiple tags with inconsistent information. It isn't clear which one iTunes will give preference to.

iTunes cannot play .wma files natively, but it can transcode them on import. If you let it do so the original are left untouched and it is the new copies that are added to the library in the standard place as determined by the tag data. You may choose the file type and bit rate from those that are available for the new copies. You are not limited to AAC. Then is no point changing from one lossy codec to a higher quality alternative since you get bigger files with no actual improvement in quality.

iTunes doesn't actively monitor the media folder (apart from Automatically Add to iTunes) so if anything else (such as Windows Media Player) moves files around this can break iTunes link to the track. If files have been deleted from iTunes, or simply placed in the media folder without adding them to iTunes, then iTunes won't see them. However if a new version of file is added to the library that should be stored in the same place as a file that already exists iTunes will make a new file but add a trailing " 2" or " 3" to the filename rather than pick up the existing file.

With respect to duplicates iTunes is sensitive to where the file is located with respect to the media folder and the options for Keep & Copy when it is added. If you repeatedly import content located outside of the media folder when copy files is active, or subsequently consolidate files and reimport, you are instructing iTunes to make duplicates. It isn't smart enough to know better. Importing existing playlists when the imported files may be copied or moved as a result of the import can also lead to problems. This may well be why some people get duplicates when importing the media folder on a new computer and they have only copied over the media folder rather than a complete working portable library.

Occasionally iTunes will put files in odd locations such as the desktop or the root of the C:\ drive if there are permissions problems with the folders where it wants to put things. See Repair security permissions for iTunes for Windows for a fix if needed.

I assume that hexadecimal encrypted folders refers to the arrangement of folders and files within the Album Artwork folder. This is an artwork cache and is required for the normal operation of iTunes. If you mean something else please say.

Now for some boilerplate....

Apple's official advice on duplicates is here: Find and remove duplicate items in your iTunes library. It is a manual process and the article fails to explain some of the potential pitfalls such as lost ratings and playlist membership, or that sometimes the same file can be represented by multiple entries in the library and that deleting one and recycling the file will break any others.

Use Shift > View > Show Exact Duplicate Items to display duplicates as this is normally a more useful selection. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks, however this works best when performed immediately after the dupes have been created. If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the recycle bin.

Use my DeDuper script (Windows only) if you're not sure, don't want to do it by hand, or want to preserve ratings, play counts and playlist membership. See this thread for background, this post for detailed instructions, and please take note of the warning to backup your library before deduping.

(If you don't see the menu bar press ALT to show it temporarily or CTRL+B to keep it displayed.)

The most recent version of the script can tidy dead links as long as there is at least one live duplicate to merge stats and playlist membership to, and should cope sensibly when the same file has been added via multiple paths.

tt2

When you add a bunch of files to iTunes it looks at the filepath of each file. If there is already an object in the library with that filepath then the new item will be ignored without creating a duplicate. Duplicates can be created if, for some reason, iTunes isn't correctly recalling the filepaths to existing content. This can happen if the designated media folder isn't available to iTunes when it is launched. Connecting the media folder won't repair all of the links that iTunes currently "knows" are broken, so it may create duplicates if asked to scan the media folder for new content. If you add content from outside the media folder and tell iTunes to make copies of anything added inside the media folder then it keeps no record of the paths where media has been imported from. Using the same folder as source for media later will create duplicates. If you rip the same CD in two different formats iTunes will see these as exact duplicates on the basis of metadata, but it won't, for example, know that one is better quality, or that you made the second one to add to an mp3 only media player. An automatic rejection of something that appears to be an exact duplicate might not suit every purpose.

Use iTunes Feedback to make suggestions for improvement.

tt2

Without looking at the content of your drive before you attempt to add it to iTunes, and not knowing what settings are in place, it is hard to be certain what the issue might be. For example there could be multiple tracks in the source folder, which is why adding them makes them show up in iTunes. Adding an XML playlist at the time of the import might also cause problems though I've never tested for this. However there should be no need to add media to a new library. If you have properly copied over your content from a previous computer then you should be bringing over the complete working library with all of its ratings, playlists, play counts, etc. in place.

In general you would copy the entire iTunes folder from the user's music folder on the old computer into user's music folder and it will just work. For those who have split out the media to a new location there are steps in Make a split library portable which may be needed to make sure that the library is still able to see the media after the transfer.

tt2

Like any other piece of software it works the way its programmers have told it to. That's not always exactly what they intended to tell it to do and, more often than not, not exactly the way you would like or expect it to.

Things to bear in mind:

1. If you have iTunes set up to Copy files to iTunes Media folder when adding to library (default option) and then repeatedly import a folder that is outside the iTunes Media folder into the library, it will do as it is told and create a new set of copies of all those files.
2. If you add a folder to iTunes which already has a bunch of duplicate files in it, iTunes will add in all the duplicates
3. If you add in untagged files then iTunes will lump them all into Unknown Artist/Unknown Album and set the track name to <Filename>.

Solutions:

1. Change your workflow. Move new stuff inside the iTunes Media folder before adding it to iTunes or disable the Copy to... option.
2. Manually dedupe before importing or see my DeDuper script for use after the fact.
3. Disable the Copy to... option and use my TagFromFilename script or fix the tags before importing into iTunes. (MediaMonkey's AutoTag From Filename is quite good for this).

tt2

I, too, am having an 'Unknown Album' crisis, but slightly different to yours.

I signed up to iTunes Match earlier in the year and I noticed today that it had matched music that was in my library with files names like Track 1, Track 2.
To find out what tracks they were, I was able to use the 'Remove Download' function, download the song back into the library, then use the 'Go to iTunes Store' to find out what the music was. The annoying thing (as far as I can tell) is that even though iTunes knows what the track is, there's no way to update the info automatically. In an attempt to give it a nudge, I basically checked every preference that required some sort of communication with the iTunes store - I think it was just Automatically Download Album Artwork and the one about sharing library information. Match already had that so what could be the harm?

So....iTunes can now not locate a significant portion of my library. Luckily I had exported (parts of) my library data and I figured out that it was only files that were in an 'Unknown Album' folder of their respective artists, but not every 'Unknown Album' folder was affected. This is what I think happened, based on changes to file locations and sizes.

• iTunes has gone searching for album artwork and found what it thinks is a match.
• If I don't have any album info on a match that been found, for whatever reason iTunes is completing that based on the artwork.
• iTunes is then automatically updating it's file structure to reflect the changes made but, for some reason, it's not updating the location information.
• iTunes then can't find the file.

I started manually relocating the files, but that was excruciating. What I've been doing instead is deleting the file it can't find via 'Remove Download' and then downloading the tracks again. The downloaded tracks are about a MB larger than the original files, which I'm assuming is because they now come with artwork! Seems to be working so far...

The new problem I have is that I now have duplicates of a lot of music on my hard drive but they're not in my library. If anyone from Apple is listening, what's the best way to put the duplicates back into the library so I can remove them without having to trawl through each of the folders? Also, some of the artwork matches are waaaaaaaaay off - so it basically thinks that one song is something completely different. I'm worried this is going to affect my uploaded music in the cloud.

But in short, I don't have an answer to your problem, but I think your hypothesis that iTunes is this problematic and non-functional is correct. Good luck with your issue.

"Looking more closely it seems you are saying that everything should be moved into the iTunes media folder"

Not exactly - this is the structure that iTunes uses, and within which it will place media files when configured to do so.

"but that is what is causing duplicates for the users above."

Yes - if it is the users that are moving files. Allowing iTunes to manage media files and folders itself is (IMO) a much more reliable approach.

"Especially if you aren't manually moving files from one computer to another or one device to another like Zune (Windows product) to iTunes."

Shouldn't be a problem - I routinely move media files between my main iTunes system and a Surface tablet (which uses the Xbox Music client - or whatever its called this week). The key processes are:

• when moving/copying to iTunes, use any of iTunes' capabilities to add media to its library, allowing it to manage files and folders itself
• when moving/copying from iTunes, use Windows/File Explorer to copy files from the iTunes media folder and then use the facilities of the other tool to add them (which for Xbox Music is just copying them to the right place).

I also routinely replicate my master iTunes library to two other systems (one at home, one at work) - again, by following a well-defined process, duplicates just don't arise.

"The only thing I would say is you probably have duplicated music in your "automatically add to itunes" folder"

No - with a correctly configured system that folder is always empty, except for the few seconds it takes for iTunes to move any media files placed there to their correct locations.

"as well as your "music" folder under itunes media, which is a waste of space unless you physically have deleted those files yourself, which is part of my steps above...."

No - why would there be duplicates unless you've moved/copied files there yourself? My iTunes Media\Music folder currently contains just over 75,000 files. I never take any actions to remove duplicate files (since I'm confident that there aren't any) - running a duplicate file finder confirms this.

"You had asked what I meant by encrypted hidden files, that is what iTunes uses to populate it's database"

Not sure how you make that conclusion. The iTunes database is populated using a combination of:

• metadata embedded in media files ("tags")
• metadata entered by the user (Get Info window)
• metadata imported from an internet resource (the GraceNote service that returns metadata for CDs to be imported)

Some of iTunes' files are not immediately readable (the database file, among others, uses a binary file format) but there's nothing hidden or encrypted.

"and it is the reason all of it's advanced features under preferences are redundant"

No - some of the settings work in conjunction with others, and some only affect a subset of operations (e.g., the "Copy files ..." selection only affects two of the five ways to add media to iTunes) but otherwise they work exactly as intended, and as described in one of tt2's earlier responses.

Bro, that is FROM the script.... I think we are done here.... Use my simple solution or try to go through that mess you are suggesting, it is up to the user:

1. Backup your music to an external drive or another location on your physical computer drives.
2. If you already have double files, use a Duplicate file remover such as tt2's script DeDuper or another program such as Digital Volcano's Duplicate File Remover Pro. iTunes could, but never will make this feature. Note: Deduper will go into the iTunes files and the following steps will not be necessary after setting up the program inputs.
3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.
4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.
5. Close iTunes
6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes
7. Open iTunes and wait for everything to populate.
8. Change any settings you would like to use under preferences > advanced from then on, but follow steps 1 to 6 for new stuff.