how to delete duplicated songs all at one time?

dbois wrote:

probably my lack of tecnical expertise, but when I hit delete I don't think the underlying file gets deleted?

If the file is inside the iTunes Media folder and you delete it from say the Music source (rather than a playlist) then you will get an option to delete the underlying file. If deleteing from a playlist Shift-Delete will offer to remove it from the library, and then (if inside the media folder) ask whether it should move the file to the recycle bin. Ctrl_Shift-Delete will just ask the last question. If a file is outside the media folder iTunes will never delete it should you remove it from the library

My DeDuper script recycles all files it sees as physical duplicates (triplicates etc.) regardless of their location.

tt2

A more useful feature can be found if you hold down the SHIFT key and then select File > Display Exact Duplicates.

uringtest2 is spot-on. I'd only add that the correct Key-Click combination for "Display Exact Duplicates" under iTunes v10.5.2 (11) is to hold down the OPTION key and then click the File menu.

You did notice that the original poster appears fo have lost interest in the problem having posed it last August didn't you?

The steps you give above are all well and good when talking about a handful of files, but no use if the aim is to accurately remove just the 5,000 accidentally duplicated files from a collection of 20,000...

tt2

Clarifications:

1. Thsi reply is for cases when an album is inadvertently imported twice.

2. I found the “Display Exact Duplicates” to be unreliable in this specific case because it also finds exact duplicates that are from different albums, which clutters the list.

Thanks, very usefull. I have displayed the tracks as an album list. That shows the duplicates in each album, and following your advice with "exact duplicates" it is much easier to get rid of the duplicate files.....Many thanks.

Don't worry it was just an observation. Threads tend to get locked automatically once the original poster has marked an answer as correct and nothing else has been added for a while. It is just that with dormant threads it is probably not that much use, at least as far as the OP is concerned, to suggest yet another alternative solution.

tt2

Hi

I used DeDuper few months ago, and thereafter i got iTunes Folder Watch as you advised me.Now, i still have some 1900 duplicates but DeDuper won't open, snd how to use it with windows to delete All my Duplicates songs ?

Check you have the most recent version of the DeDuper script from my site. It should be version 1.0.1.2 dated 2nd December 2011.

Please note, the script is only intended to work on things that iTunes would treat as "Exact Duplicates" and it only works on files that are connected to the iTunes library. Before starting all files in your media folders should be added to the iTunes library.

In the context of this thread, type 1 or logical duplicates are multiple entries to the same file and type 2 or physical duplicates are multiple copies of the same file. Type 2 dupes will have identical tag information (at least they will unless you edit one of them) and the same file size thus both types will display as "Exact Duplicates" if present.

The script avoids the more complex problem of trying to detect and selectively remove what I earlier called type 3 duplicates, that is alternate recordings or rips at different bit rates of the same tune, principally because that is a much harder problem to automate, but also because "best track to keep" from a set of potential candidates will be a subjective matter.

tt2

I'm finding this thread interesting as recently I offered to sort out a friends itunes library which had been backed up into three different folders on an external drive on three separate occasions.

He had many duplicate files. But I managed to merge them all and I was left with just duplicate entries - rather than actual files, which I couldn't seem to solve - as described by the original question in this thread.

When I selected display exact duplicates, at this stage, it would produce "duplicates" only - so I was a little baffled.

However, all this is on a macbook running leopard OS. I see this thread is mainly about Windows. Is there a script you have written that will work for mac?

thanks

curlydog

When you ask iTunes to Display Exact Duplicates, the legend at the bottom of the screen still changes to Displaying Duplicates though you should be able to tell that it is doing the job properly by comparing the number of results relative to the number given by Display Duplicates.

Sadly I don't have access to a Mac, but there should be something useful at Doug's Scripts.

If Date Added, Ratings, Play Counts etc. don't matter then you could delete all the entries from the library, without deleting the underlying files, then reimport the iTunes Media folder to restore all the tracks that have been removeed. (N.b. don't do this if there are files in wav format as these will lose associated information.)

Or you can go old school and CMD(or is it option?)-click all but one of each group of repeated tracks, a screenful at a time and delete them that way, again keeping the underlying files.

tt2

2 relatively stupid questions for turingtest2, with the preface that my username is "Lost in an itunes fog", if thats any indication...

I have 2 major libraries of songs, one on an external, on on my PC. Each is about 35GB. I have a ridiculous amt of duplicates and need to get rid of a lot, like a few GB's worth...

1. If I go into the itunes music folder via C: drive and delete one-by-one all of the songs that are duplicates, then re-load this into my itunes library, there wont be any duplicates, right? In addition to this taking forever, as you stated previously, if these files do not say 'duplicate', is there any reason that I can't delete 2 of the 3 per say and keep the one, with the assumption that it will function?

2. If I'm in itunes and do the same systematic deletion of 2 of 3 identical songs, leaving one behind, I have no way of knowing whether its a copy of not aside from look at the "get info", but the moral of the story is that if I delete these 2 of 3 to my recycle bin, will that 1 remaining function fine as a copy? Is there something wrong with deleting the 'original' and keeping the 'copy'? Will they play, because they appear to be at the moment?

Finally, I used your DeDuper script on 1500 tracks and it cut out about 150 of them as duplicates. However, when I went back and looked via the "exact duplicates" method on itunes, it's still showing a ton of duplicates. Does that mean these are the type 3 dups?

Bottom line is I'm willing to take the long road to get rid of all of my duplicates, if that means just scrolling through my Itunes media folder and manually deleting 2 of the 3 or so dups, but is there anything wrong with that? Ie is it possible that if I delete the original, then that is no longer playable?

Thanks

1. If you clean up visible duplicates via Windows Explorer then you will create missing file links in iTunes. These can be removed with the aid of a script or iTunes Folder Watch or you can delete the contents of your library and reimport. Note starting a fresh library will have implications for any devices synced with the current library as iTunes would want to remove all data from the device and start over.

2. As long as you can be sure that you have eliminated logical/type 1 duplicates then you are free to remove other redundant copies without breaking the tracks that you want to keep. If iTunes lists three tracks as exact duplicates then the tag information between them tallies and it is safe to delete two of them. If you expose the size and kind fields in your view you can choose whether to preserve the largest/best quality, smallest, or favoured format.

Yes, if you have used my DeDuper script then any remaining exact duplicates should be what I have described as type 3 duplicates, that is different rips of the same song in different formats or bit rates. I have, as yet, ignored this group because I couldn't think of an obvious way to determine the "best" file to keep in these circumstances since different people might have different views on what is best for them. That said I suppose it wouldn't be too hard to extend the script with an option that preserves either the largest or the smallest file, or leaves them all as it does currently.

Again, as long as there are no logical dupes then you can cull from the remainder without accidentally breaking existing track entries. If you are prepared to wait a few days I'll see what I can do about extending the script further. What are your thoughts on which file should be preserved from the dupes? I'm leaning towards largest on the grounds that it should be the best quality, leaving people free to downsample later if they want to reduce file size rather than forking out for a bigger drive.

tt2

turingtest2 wrote:

The script avoids the more complex problem of trying to detect and selectively remove what I earlier called type 3 duplicates, that is alternate recordings or rips at different bit rates of the same tune, principally because that is a much harder problem to automate, but also because "best track to keep" from a set of potential candidates will be a subjective matter.

Having given this matter some more thought I've now rewritten the script to take this last group of duplicates into account. I've decided that in most cases the largest type 3 or alternate duplicate will be the one that you want to keep, but I've made this part of the new script fully optional so that those who prefer to check their alternate versions more closely can stll make use of the script.

The latest version of DeDuper also adds a feature to preserve playlist membership, though not necessarily the precise position. If a "dupe" is removed from a given playlist the "original" is added to the playlist unless it is already a member of it.

tt2

Hi tt2,

Thank you for you for your posted help so far. I had a problem error message thouhg pop[ up interrupting DeDuper. I got a message noting "error: file is unassociated"

I don't believe i've seen a fix for this otherwise, and this issue is preventing me from eliminating my duplicates with the benefits you designed into DeDuper.

Thanks,

Hmm, do you have many broken entries in your library? Tracks that show up with exclamation marks? I thought the script was set up just to ignore them but I didn't specifically create and test any under the latest build. Will look at it properly tomorrow, but the precise error message would be useful.

tt2