Q: Why does iTunes duplicate, triplicate, or even quadruple songs I import into it?

Itunes duplicates playlist items.  I'm using 12.2.2.25 at the moment and copying my music to a new computer.  I can import a folder with single copies of songs. I export the playlists from the old computer. I then import my playlist into my new computer.  Itunes makes a duplicate copy of the songs on my computer that are in the playlist.  If the song is used in multiple playlists, the song is duplicated many times.   If you use itunes duplicate removal method (view, show duplicates, select all, delete), the imported playlists are broken. 

This is a real pain for those of us who use complicated play lists when we upgrade computers. 

You should clone the entire library, then there is no need to import anything, and no risk of creating duplicates.

See this migrate iTunes library post for suggestions on copying the library from your old computer to a new one.

tt2

An external drive is the worst recommendation for a permanent location to be reading and writing music files. The amount of reads/writes listening to music would tax on an external drive is just ASKING for a drive failure. External drives are NOT designed for this unless you have purchased a very high end system configured for stability. I recommend no one attempt to read their large collections of music off of an external drive.

Additionally, I am going to second this issue and say I am confounded at the options and the results of the file storage for each setting.

"Keep folders organized" - created a music folder under my iTunes media folder and placed only a handful of artists splitting my music up and create duplicates there.

"Copy files" setting - Turned off when importing after moving everything to a music folder under iTunes Media folder I somehow still had duplicates so that feature isn't working.

"Changing iTunes media folder location" - You would think this is where the music should be, but the "iTunes Library" file is a directory above which created another set of duplicates: this is where I am seeing quadruple duplicate files. I'm going to see what deleting this file altogether does. Maybe it will trick itself into only seeing the music files once.

Placing files into "automatically added"- why would I put files into a folder that would cause the program to copy everything over and then run into the same issues of reading everything doubled up?

I'm assuming here that the hexadecimal encrypted folders that keep popping up when I try to import have something to do with it. If iTunes needs to read files that way, it is highly possible it cannot tell the difference between something it converts into it's special code and something that all other media players can normally just read. It's not like I can just switch either. I'm stuck with 3,000 purchased iTunes songs. A check and a stack of CD's would be nice at this point Apple....

Window shell shows all information is tied to the mp3, mp4, .flac, WAV whatever extension. -> iTunes places some music in at least 10 different folders named unknown artist. iTunes is clearly not designed to read music files other then their made up AAC extension

on a side and unrelated note iTunes tried to convert unsupported extensions to AAC. If I purchased music I don't need itunes to completely destroy the codec i purchased.

SIMPLE SOLUTION:

So, based on an amalgamation of the posts above I came up with a stupidly simple solution:

1. Backup your music to an external drive or another location on your physical computer drives.

2. If you already have double files, use a Duplicate file remover. iTunes could, but never will make this feature.

3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.

4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.

5. Close iTunes

6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes

7. Open iTunes and wait for everything to populate.

You should only see the exact files that you want in iTunes. If you still have duplicates you actually physically had duplicate files to begin with and you need to adjust how you remove them in the Duplicate file remover software such as "remove similar names" because "01 - Song" could be the same as "1-Song".

After your music is all loaded up you can delete everything out of the "Automatically Add to iTunes" folder. Unless you want to physically have 2 copies of the music on your drives.

The sad part about this solution is that in order to stop showing duplicate entries of music in iTunes you actually have to physically duplicate files. So, make sure you have enough space on your hard drive to temporarily store double the amount of data, otherwise it is impossible for you to use the iTunes program until you purchase a bigger hard drive.

The reason iTunes is showing the duplicate files is because if you didn't do it this way, and keep trying to adjust settings or add music directly to the folder that iTunes ends up creating for music, the library database keeps growing with duplicate entries. When iTunes goes to read what on the database it thinks there is supposed to be double of everything. It would make sense to just put the music in the folder it is supposed to go in, instead of a different folder that isn't very descriptive at all, but alas nothing makes sense in the Apple world.

Every external drive I've bought has got the same kind of 3.5" or 2.5" hard drive as is fitted internally to the desktop or laptop computers that I have bought. Internal is best if you have the room but a locally connected external works fine. A backup is vital. All drives will fail eventually, often without warning. The key factor is large. If the library is large enough not to fit on what is inside your box then it needs to go outside. A split library with media on NAS storage is a possibility, but iTunes doesn't play well with NAS and transfer rates are generally lower than with a local drive. RAID is great for those with the budget for it.

Let's try to explain the terms.

Keep organized controls what happens to files that are located inside the media folder when they are added to the library or if you make a change to their metadata after they have been added to the library. If the option is enabled the folder and file names may be edited to reflect the content of the tag, with music typically going to <Media Folder>\Music\<Album Artist>\## <Name>.<Ext> otherwise the file stays where it was. (There may be an optional leading disc number if the disc number or disc count is greater than 1, Artist is used if Album Artist is blank, and there is also an iTunes specific compilation flag that can put tracks in <Media Folder>\Music\Compilations<Album>.)

Copy files controls what happens to files that are outside the media folder when they are added to the library. If it is enabled iTunes makes a new copy inside the media folder based on the tag properties. It doesn't "remember" that the original has been imported so you are at risk of making duplicates if you import the same folder again. If the option is disabled iTunes connects to the file on its original path, the file's tag will be updated if you edit properties, but the file won't move.

Consolidate files tells iTunes to import files that are currently outside of the media folder in the same way that it would if they were added when Copy files was active. In and of itself consolidate files doesn't create duplicates within the iTunes library, but there is the potential for you to reimport the original files later unless these are actively archived or deleted after consolidating.

Anything ripped from CD, downloaded from the iTunes Store, converted to a new format, or added using the Automatically Add to iTunes folder is moved into the standard folder structure whether or not Keep organized or Copy files are enabled. A typical use case for Automatically Add to iTunes would be if you had a drive with media brought over from another computer that you want to add to iTunes. You can either add the folder when Copy files is active, or copy the folder of media into Automatically Add to iTunes. You shouldn't ever need to use the folder as part of moving an existing library from one computer to another. An advantage of using the folder will be that anything that iTunes cannot handle will be moved into a folder called Not Added. if you just add the folder in the normal way iTunes will silently ignore any track it doesn't like but you will be none the wiser.

Changing the media folder location tells iTunes where new media will be stored. When you make the change iTunes will typically offer to consolidate files to the new location, but you don't have to let it do so. If you want to change which library is opened you need to hold down shift (Windows) or atl/option (Mac) as you launch iTunes and then use the options to choose a library or create a new one.

Apple don't refer to the library folder as such, but it is the folder that contains the library database file, normally called iTunes Library.itl, along with some supporting files, the Album Artwork cache folder and, typically, the media folder iTunes Media. When all content connected to the library is inside the media folder, and the media folder is directly inside the library folder, then the library is in a portable shape that can be easily moved from one computer, drive, or path to another with out breaking the links between the library and the media. More at Make a split library portable.

Files in .wav format do not carry a tag. Windows Explorer shows no additional information for them. iTunes will import these to the Unknown Artist\Unknown Album folder. If they are still on paths that reflect Artist and Album after they are added to the library (which will depend on original location, media folder location and choices for Keep & Copy) then I have a script that will read in the data from the path: TagFromFilename. If cannot help if the files have been moved into Unknown Artist\Unknown Album unless there is an alternate undamaged source.

Occasionally .mp3 files have multiple tags with inconsistent information. It isn't clear which one iTunes will give preference to.

iTunes cannot play .wma files natively, but it can transcode them on import. If you let it do so the original are left untouched and it is the new copies that are added to the library in the standard place as determined by the tag data. You may choose the file type and bit rate from those that are available for the new copies. You are not limited to AAC. Then is no point changing from one lossy codec to a higher quality alternative since you get bigger files with no actual improvement in quality.

iTunes doesn't actively monitor the media folder (apart from Automatically Add to iTunes) so if anything else (such as Windows Media Player) moves files around this can break iTunes link to the track. If files have been deleted from iTunes, or simply placed in the media folder without adding them to iTunes, then iTunes won't see them. However if a new version of file is added to the library that should be stored in the same place as a file that already exists iTunes will make a new file but add a trailing " 2" or " 3" to the filename rather than pick up the existing file.

With respect to duplicates iTunes is sensitive to where the file is located with respect to the media folder and the options for Keep & Copy when it is added. If you repeatedly import content located outside of the media folder when copy files is active, or subsequently consolidate files and reimport, you are instructing iTunes to make duplicates. It isn't smart enough to know better. Importing existing playlists when the imported files may be copied or moved as a result of the import can also lead to problems. This may well be why some people get duplicates when importing the media folder on a new computer and they have only copied over the media folder rather than a complete working portable library.

Occasionally iTunes will put files in odd locations such as the desktop or the root of the C:\ drive if there are permissions problems with the folders where it wants to put things. See Repair security permissions for iTunes for Windows for a fix if needed.

I assume that hexadecimal encrypted folders refers to the arrangement of folders and files within the Album Artwork folder. This is an artwork cache and is required for the normal operation of iTunes. If you mean something else please say.

Now for some boilerplate....

Apple's official advice on duplicates is here: Find and remove duplicate items in your iTunes library. It is a manual process and the article fails to explain some of the potential pitfalls such as lost ratings and playlist membership, or that sometimes the same file can be represented by multiple entries in the library and that deleting one and recycling the file will break any others.

Use Shift > View > Show Exact Duplicate Items to display duplicates as this is normally a more useful selection. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks, however this works best when performed immediately after the dupes have been created.  If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the recycle bin.

Use my DeDuper script (Windows only) if you're not sure, don't want to do it by hand, or want to preserve ratings, play counts and playlist membership. See this thread for background, this post for detailed instructions, and please take note of the warning to backup your library before deduping.

(If you don't see the menu bar press ALT to show it temporarily or CTRL+B to keep it displayed.)

The most recent version of the script can tidy dead links as long as there is at least one live duplicate to merge stats and playlist membership to, and should cope sensibly when the same file has been added via multiple paths.

tt2

A few follow-ups to tt2's comments:

• I agree about the suitability of external drives - I've been using these for iTunes for the best part of a decade and not encountered any failure rates higher than internal drives.  The one caveat that I would add is that "desktop" drives (larger form factor and with dedicated power supplies) are more reliable than "portable" drives that draw power over USB.  As tt2 says all drives fall eventually, so a strategy for making and maintaining backups is essential.
• For those who like pictures this is the standard layout of iTunes media folders:
  
  I've yet to encounter a convincing argument for using a different structure or not allowing iTunes to manage where media is stored.
• Apple really should put something in the UI that states that the iTunes Media folder location setting in Edit > Preferences > Advanced does not change where iTunes looks for the existing content of your library.
• The interaction between the Copy files ... option in Edit > Preferences > Advanced and the different ways of adding media to the iTunes library can be summarized as follows:
  

Thanks for the settings definitions tt2. They reiterate your point earlier in the thread, and mine as well. Each setting does what it does, and you tell iTunes to either do them or not. However, there is no reason for someone to think through all of that on their to come up with the solution that I have already given above. Of course iTunes has an official solution, but that is useless when you think about the work of manually selecting everything. My approach is a third party duplicate remover that has a much more user friendly interface, but thank you for providing some source code. iTunes could begin supporting batch and control files that do all of this for one folder on windows, but like I said they won't.

You get what you pay for with external drives. The $60 range is going to fail 10 times more then the $130 range, but I would never buy something that big or expensive when I want portability. For portability and space your price skyrockets. https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/

NAS your talking even more money, which is also starting to get a little off topic here. The point is your suggestion is not advisable for a little Toshiba that someone uses to move large files, and have sit on a shelf the rest of the time and you have to really get into expensive and non portable alternatives to be reliable.

Thanks for the extra directory hierarchy httg. It is a good point that iTunes doesn't just duplicate, triple, or even quadruple songs, it does everything else too.

tt2 I've also considered your notes on the encrypted files and it seems there is more to them then artwork. If I were to copy songs directly off of an ipod, I wouldn't see names to these songs like a normal song file, I would get a hex based name under a set of hex named folders. Dropping them all into iTunes and playing them would decode the file and generate a copy in the iTunes media folder (assuming you copy these files off of an ipod to some other location). So, I am assuming that in order for the iTunes library.itl to talk with a number of apple devices, the database must store a hexadecimal data entry that can be decoded by each device. Since the HEX will be unique for each file it is impossible for iTunes to determine that they are the same or similar file, thus rendering the features of "Keep organized, consolidat, copy files etc." useless for all but one computer that has only iTunes bought songs on it. Additionally over time these hex entries in the database will just grow and grow and grow uniquely every time you copy stuff over or continue to use the settings when you get a new computer. You might have 2 physical copies of a song, or even one single copy, but once the Library.itl assigns a hex code to it, it has to generate a new one for it to consolidate, and do all that other stuff.

The point is there is only one way to do it quickly and effectively as I've listed above using a  third party cleaner or running your source code, and never using the other settings again. Every single one of those settings and features has a potential to duplicate files, so none of them should be used because once it starts to multiply it's worse and worse until someone cuts it off at the root on a new build with a completely new library.itl.

"It is a good point that iTunes doesn't just duplicate, triple, or even quadruple songs, it does everything else too."

No, it doesn't - in nearly a decade of using iTunes and (once I'd decided to put my entire CD collection into it) build up a moderately large library (4500+ albums) I've never once seen a case where iTunes duplicated anything, except in cases where I'd mistakenly imported media file(s) that were already present in the library - and I doubt I've done that more than a handful of times.

Okay, I guess I missed what your point is with the illustrations. Looking more closely it seems you are saying that everything should be moved into the iTunes media folder, but that is what is causing duplicates for the users above. Also, what you are showing is for an apple product. Assuming you are using apple products exclusively all these years, I would say that you aren't going to run into these issues. Especially if you aren't manually moving files from one computer to another or one device to another like Zune (Windows product) to iTunes. Whenever you sync or plug in from one apple product to another it knows what to do because they all talk to each other off of the itl and xml files. Where the issue arises is a physical copy from one drive to another with something like an external drive.

This discussion is great and all, but there is a step by step solution  to the original issue above.

"Looking more closely it seems you are saying that everything should be moved into the iTunes media folder"

Not exactly - this is the structure that iTunes uses, and within which it will place media files when configured to do so.

"but that is what is causing duplicates for the users above."

Yes - if it is the users that are moving files.  Allowing iTunes to manage media files and folders itself is (IMO) a much more reliable approach.

"Especially if you aren't manually moving files from one computer to another or one device to another like Zune (Windows product) to iTunes."

Shouldn't be a problem - I routinely move media files between my main iTunes system and a Surface tablet (which uses the Xbox Music client - or whatever its called this week).  The key processes are:

• when moving/copying to iTunes, use any of iTunes' capabilities to add media to its library, allowing it to manage files and folders itself
• when moving/copying from iTunes, use Windows/File Explorer to copy files from the iTunes media folder and then use the facilities of the other tool to add them (which for Xbox Music is just copying them to the right place).

I also routinely replicate my master iTunes library to two other systems (one at home, one at work) - again, by following a well-defined process, duplicates just don't arise.

iTunes stores the media files on an Apple device in an obfuscated fashion, using semi-random four character file names with the original file extension in a number of hidden folders. This is principally an anti-piracy measure. Apple have made it hard for somebody to load up their device with media, then take it around to a friend's and copy off selected albums. If you manage your library and its backup properly there is no need to delve inside the storage system on a device. They are not intended to be used to transport media from one library to another.

Duplicates in iTunes are always the result of the user making invalid assumptions about the consequences of some action which effectively instructs iTunes to make the copies. Using a Windows (or Mac) file deduper, then deleting the library database and reimporting works after a fashion, but throws away ratings, play counts, playlists, date added information, all metadata related to .wav files, etc. My script aims to avoid all of those drawbacks.

tt2

I don't know why that helps anyone here by saying that you haven't had issues. It sounds like you've followed all of the steps above when managing your music which is why you don't have duplicate files. The only thing I would say is you probably have duplicated music in your "automatically add to itunes" folder as well as your "music" folder under itunes media, which is a waste of space unless you physically have deleted those files yourself, which is part of my steps above....

You had asked what I meant by encrypted hidden files, that is what iTunes uses to populate it's database, and it is the reason all of it's advanced features under preferences are redundant. There was a time when your ipod was potentially the only remaining stored location of your music and you had to go this route to rebuild your library. I don't know why you keep saying use deduper, that or another program would work the same...it's up to the user:

1. Backup your music to an external drive or another location on your physical computer drives.

2. If you already have double files, use a Duplicate file remover such as tt2's script DeDuper or another program such as Digital Volcano's Duplicate File Remover Pro. iTunes could, but never will make this feature.

3. Delete the iTunes folder which includes iTunes Media, iTunes Library.idl, genius, and extras.

4. Open iTunes and set your preferences to only look in the location you want to store your music, don't have it do anything else because none of the other features work. Edit > Preferences > Advanced > iTunes Media folder location at the top. Uncheck everything except "Enable full keyboard navigation" unless you don't want to use your keyboard to shuffle your songs.

5. Close iTunes

6. Copy your music into the "Automatically Add to iTunes" folder. It is located under the path you just set to store your media. iTunes\iTunes Media\Automatically Add to iTunes

7. Open iTunes and wait for everything to populate.

8. Change any settings you would like to use under preferences > advanced from then on, but follow steps 1 to 6 for new stuff.

Why would anybody want to use steps one though six every time they add some content to the library? Hopefully this isn't a situation anyone gets into more than once. Moreover most people will want to preserve as much of the effort previously invested in building their library as possible when deduping which is why I want to highlight the particular drawback that your suggestion to use a file based method and then rebuild the library from scratch results in. If sombody uses the DeDuper script at step 2, then 3 through 8 are redundant.

This comment:

don't have it do anything else because none of the other features work.

suggests that both hhgttg27's and my efforts to explain how those features work and exactly what their impact is have failed.

tt2