12 Replies Latest reply: Sep 18, 2013 5:47 PM by turingtest2
pet2000 Level 1 Level 1 (0 points)

Hello,

 

I have a PC running windows 7.  I keep all my music in one folder on a NAS device.  I am getting more and more fed up with Windows Media Player (WMP) and I would like to move to iTunes.  This move, however, appears to be as painful as using WMP.  Please help if you can.  I have used iTunes before so I have an existing music and video library and I also have a number of playlists both with music and videos.  I cannot afford to start from sratch, i.e., to remove and readd everything; it would be too costly.  I have two major problems with iTunes:

 

1. When I try to add files that already exist in the iTunes library iTunes happily adds these file and creates duplicates.  I cannot find a knob that would prevent iTunes from creating these duplicates and simply ignore any found files if they are already present in the library.  All such duplicates point to the same file on disk.  I am really scratching my head over this as I would think that this would be a very basic thing to do/allow, yet it does not seem to be possible.  Is it me missing something simple here or is it Apple telling me that this is how it must be...???

 

All I want to do is to add the top folder that contains all my music without having to deal with thousands of duplicates.  I cannot add the files one-by-one because there are too many of new files there and if I remove all the files and re-add them all my lists will likely become empty (tried it on one list and, as expected, the list got emptied once I removed the files that were in the list).  Hmm, is this asking for too much?  Please help.

 

2. I do not seem to see a way to tell iTunes to monitor for new media files.  WMP allows you to specify folders to monitor.  You add a file to the folder and the file gets added to the library.  What is wrong or bad with this concept (other that it would not work if duplicates were handled)?  Why does iTunes not seem to support that?

 

Thank you,

Peter


iTunes, Windows 7
  • turingtest2 Level 9 Level 9 (61,330 points)

    iTunes only creates duplicates if you ask it to, for example by adding the same folder of content twice when that folder is outside the iTunes Media folder and iTunes is set up to copy added music into the media folder, or if the folder you are adding already contains duplicates. A have a script called DeDuper for clearing duplicates from the iTunes library. See this thread for background.

     

    If you want to monitor a set of folders in the WMP fashion use iTunes Folder Watch.

     

    See also Grouping tracks into Albums and  Getting iTunes & Windows Media Player to play nicely for some more background reading...

     

    tt2

  • pet2000 Level 1 Level 1 (0 points)

    tt2, thank you for your reply.  Let me restate your answers the way I see them.


    Problem #1:  You said that "iTunes only creates duplicates if you ask it to...".  Well  my goal is not to create duplicates but to avoid them and the problem with iTunes is that it does not seem to offer an easy way to do this.   Do not get me wrong, I like iTunes but let's call black black and white white.  So the way I read your answer is: no iTunes does not support a way to add the same folder twice without creating duplicates. 


    Problem #2: The answer to the second question is also no, iTunes does not offer any way to monitor for new files in a folder or folders.


    I wish the above were incorrect but from what I hear and from what your wrote I conclude iTunes does not support any of that.


    You offered external to iTunes solutions for both problems.  Thank you for that!  I did not know about this tools.  Having said that my iTunes music files are on a NAS device (files not copied to iTunes library), my video files are on NAS but (because of the necessity to convert to iTunes supported format) they were transcoded and stored in iTunes library, my iTunes Store purchases (music) are in the iTunes library and on the NAS (backup - likely duplicates).  In short it gets messy very quickly and after reading the NAS and multi-path warnings for the DeDuper tool I am just not jumping on the idea of using it - there would be physical duplicates and I would not want any physical files to be deleted.  tt2, do you know if the DeDuper offers a mode where it would present the user with a list of duplicates and allow the user to decide which dups to remove before doing anything?  I basically would want to remove the dups from the library but without deleting any physical files.


    I will have a closer look at the iTunes Folder Watch program.  Unless I am misinterpreting something I think it should do what I want to do. 


    So again, thank you for your answers and suggestions with the external tools.  It is just too bad that iTunes does not support what I (and many others) need natively.


    Peter

  • pet2000 Level 1 Level 1 (0 points)

    tt2, one more thing, I forgot about it.  If I understood you correctly you stated that:

     

    "...adding the same folder of content twice when that folder is outside the iTunes Media folder and iTunes is set up to copy added music into the media folder..."

     

    will create duplicates.  To my this implies that if iTunes were set up NOT to copy the added music into its media folder duplicates would not be created.  Well my music is on a NAS device and outside of the iTunes media folder, iTunes is set up NOT to copy the added files but it does not stop iTunes from creating the duplicates.  What I see in the library are two entries pointing to the same file on the NAS.

  • turingtest2 Level 9 Level 9 (61,330 points)

    If iTunes starts up when your media folder is offline then it may flag up all your files as missing. If you subsequently connect the drive and import the media folder to add files then it can add new entries because it doesn't "see" that the the files have already been imported. Shut down and restart iTunes and now both copies work. These are what I call logical duplicates. They need to be tackled by deleting all but one of each set of repeated entries from the libray only, leaving the file where it is.

     

    Another potential source of problems is when the same set of media can be accessed via multiple paths, e.g. as \\<device>\<path 1> or X:\<path 2> where X is a mapped drive letter, or if junctions/folder redirects have been used to make folders appear on different paths. C.f. the virtual remap of requests to Documents and Settings through to Users that can happen on Vista/7. Currently this type of duplicate appears to my script as physical dupes and the script tries to delete all but one, sending to the recycle bin. This is undesirable because the files are really logical dupes and there is no recycling on remote storage so the files are just deleted.

     

    At present I haven't worked out a way to automatically detect if multiple paths to a given file exist. I have however been toying with the idea of moving the dupes rather than deleting them so that they could be restored if deleting turns out to be an error. If all your duplicates show as either on the same path, or two distinct files on two independent paths, then the script is safe to use. If you're not certain (and don't have a backup) then I would not recommend you use it in the current form.

     

    tt2

  • pet2000 Level 1 Level 1 (0 points)

    I understand the issues with mapped drives and the virtual remaps.  This is exactly what "scares" me because I am not 100% sure if I have duplicate paths and/or remaps or no.  I did some remaps on my older PC and I am not sure if I did it on the newer one or not.  I reinstalled w7 recently and I am not sure what survives such a reinstall.  My main point is that I am happy with my physical files, I do not want to mess them up by having some of them deleted.  Yes I have a backup but I do not want to be forced to use it - this will only create more problems.

     

    Ideally I would like to remove any logical dups but without touching (in particular removing) any physical files.  I guess your DeDuper does not support a mode or flag that would stop it from removing any physical files... or does it?

     

    If I were you I would not try to detec multiple paths either.  Neither would I try to move physical (or any) dupes.  For me that would be too much cost for what you get.  I would consider though the idea of having a "non-intrusive" mode with respect to the physical files, i.e., no mods of any kind to the files.  But than again, I am biased as this is what would work for me... :-)

     

    Thanks,

    Peter

  • turingtest2 Level 9 Level 9 (61,330 points)

    You've given me some food for thought. I think I've got a plan but I haven't got time to edit & test a rewrite just now. With luck I can get to it later today or tomorrow. Instead of the current Yes/No/Cancel question after the files have been scanned I'll put in something regarding the max. level of duplicate to clean from the following, ideally before scanning takes place.

     

    Playlist

    Logical

    Physical

    Alternate

     

    There are also two versions at the moment, the main one keeps the largest alternate dupe, the other keeps the smallest. Hopefully I can wrap up other options into the prompt so it becomes a choice from something like this.

     

    1. Report
    2. Playlist (& 1)
    3. Logical (& 1-2)
    4. Physical (& 1-3)
    5. Alternate (Keep Largest) (& 1-4)
    6. Alternate (Keep Smallest) (& 1-4)
    7. Alternate (Keep Oldest ) (& 1-4)
    8. Alternate (Keep Newest) (& 1-4)
    9. Alternate (Choose) (& 1-4)

     

    This would all be much easier to do in some respects as a full blown app with proper dialog boxes, but keeping it as a script makes it easy for anyone else to inspect the code and adapt it if they choose.

     

    tt2

  • pet2000 Level 1 Level 1 (0 points)

    I assume that (1. Report) will do nothing  to the library or files;  it will just report the different levels of dupes found but would not modify anything.

    I am not sure what  (2. Playlist) would do.

    I assume that (3. Logical) will remove the logical dupes, that is the dupe entries in the iTunes library, no file removal (again not sure what 2 does).

    I assume that (4. physical) would remove the physical dupes (files on disk) and do the 1-3 steps.

    Putting alternates after physical makes sense to me.

     

    I think this would give the user lots of flexibility including what I was after (no changes in files).

     

    Peter

  • turingtest2 Level 9 Level 9 (61,330 points)

    The report level would be just as you say. Currently the script performs playlist & play count merges whether or not that class of dupe goes on to be removed. I'll still add in a confirmation step after the report but I can probably omit the merges for files that aren't going to be cleaned. There is probably a really good reason why I did the merges in the scanning phase (concerning what happens if you interupt the process) but it would be more elegant if it can be done later. If I'm going to do that as well that will add to the complexity of the rewrite.

     

    Playlist dupes are the same tracks from the iTunes database added to a regular playlist. This is a perfectly valid action, but can cause confusion when trying to clean up... If someone has chosen to gather together a playlist of "things that need fixing", rather than a playlist where playback order is relevant and repeats might be intentional, then it is probably worth cleaning any "playlist" only dupes out of the list while the script is tidying up, if only to provide visual confirmation that all the dupes have been removed, or perhaps more importanly, not leave any mystery ones...

     

    I hadn't even thought about the possibility of them until trying to debug the script for someone else where such duplicates caused an earlier version to bomb after removal as the list shrunk in size by two or three instead of the one that was expected.

     

    tt2

  • pet2000 Level 1 Level 1 (0 points)

    I see...  I have not thought about this kind of playlist dupes either.  But that's perhaps because such repeats in lists do not have to be treated as "real" dupes.  They could be though of as, possibly multiple, references to the same actual entry in the library and only the library entries could have duplicates.   Well, unless you are talking about list references to real dupes... Hmm, this is actually what you may be talking about. What I mean here is that there could be a dupe in the library, e.g., song1 and song1' and then in a list there could be, e.g., 2 references to song1 and 3 to song1'.  How many of such references DeDuper should leave in the list?  I think DeDuper should remove one of the song1/song1' entrires from the library and the corresponding references in the list leaving 2 or 3 refeernces in the list.  This is intiuitve (at least to me) and probably simpler than other approaches...

     

    So what is the default behavior of DeDuper?  Does it remove such dupes from playlists by default or not?  I am still not sure what the plalist level does.

  • turingtest2 Level 9 Level 9 (61,330 points)

    Reworking the script is taking longer than I thought, partly because I've been busy at work and also because the new iOS release has taken some of my attention.

     

    Because of the way iTunes provides access to the tracks for scripting purposes it is common to process lists of tracks in reverse order. That way if any update causes the item you've just processed to evaporate from the list you can still work with the rest. Playlists duplicates, where the same iTunes entity appears twice or more in a user playlist, need considering simply because not considering them can lead to errors. Previously such items would have been read by my script as logical duplicates since both connect to the same file on disc, however removing one entry to "dedupe" would have removed the track from the library, thus also removing all copies from the playlist being processed. This also causes the script to crash out because the length of the list shrinks by two (or more) while the index into the list only reduces by one.

     

    The features that I've put into the current version that merge playlist membership, ratings, play & skip counts, last played & skipped are all currently connected to the scanning phases rather than the delete phase, so I'm going to need to rework a few sections of code to make it work properly, ideally without making the script take twice as long to run...

     

    An added complication I've proposed is to decide which type of alternate dupe to keep after they have been counted (or possibly on a track by track basis). Currently the script works on the basis that logical dupes are detected from a list cleaned of playlist dupes, physical dupes are detected from a list cleaned of playlist and logical dupes, and alternate dupes detected from a list cleaned of playlist, logical & physical dupes. I'm still wrapping my head around the best way to make sure that the metadata that is being kept (ratings, playlists etc.) is applied to the correct file when I move that code into a different section.

     

    tt2

  • KnitB Level 1 Level 1 (0 points)

    The feature we want is "Look in this folder and add to iTunes any song you don't have already / haven't imported previously."

     

    Giving iTunes this feature would make it much easier for people to use content purchased from other stores, use devices from other vendors, and use different music players. Each of these works against Apple's interests. It's never going to happen. End of story.

  • turingtest2 Level 9 Level 9 (61,330 points)

    iTunes already does that just fine as long as either the folder you want it to scan is the iTunes Media folder or is somewhere inside it, OR the folder you want it to scan is anywhere, but the option to Copy files to the media folder when adding to library is not enabled...

     

    tt2