10 Replies Latest reply: Jan 17, 2013 3:52 PM by gazzabhoy
Zwockel Level 1 Level 1 (0 points)

I want to migrate a nas based music library with more than 25,000 files. I read article http://support.apple.com/kb/HT4527 and transfered the files per external drive without prior consolidating. The files should reside at the nas of course. I want to retain the directory structure of my nas because other devices (as network players) should access the audio files too. I didn't use drive letters to access the audio files with my Windows-PC, I used the unc-filename convention.

Therefore the location of a audio file is for example //nas/audio/MP3/Musik/songs/r/Ram Jam - Black Betty.mp3

 

Copying iTunes Folder

Having copied the files into the Music/iTunes folder on my mac almost everthing is fine but the music files are marked as missing ("!"). The location then is shown as file://localhost//Nas/audio/MP3/Musik/songs/r/Ram Jam - Black Betty.mp3. When I point iTunes to the location of the file it changes to /Volumes/audio/MP3/Musik/songs/r/Ram Jam - Black Betty.mp3

 

Editing iTunes Music Library.xml

I edited the iTunes Music Library.xml using TextEdit. Interestingly the location there was before

<key>Location</key><string>file://localhost//Nas/audio/MP3/Musik/songs/r/Ram%20J am%20-%20Black%20Betty.mp3</string>

and is now

<key>Location</key><string>file://localhost/Volumes/audio/MP3/Musik/songs/r/Ram% 20Jam%20-%20Black%20Betty.mp3</string>

 

I closed iTunes and replaced all occurences "file://localhost//Nas/" to "file://localhost/Volumes/" saved the file. This method worked under Windows when I moved the audio files to another nas-box. Apparantly iTunes uses the iTunes Music Library.itl file which is binary, so this did not work.

 

Importing iTunes Music Library.xml into empty iTunes

I found another link http://support.apple.com/kb/HT1451 and i tried to import the edited XML-File into an empty iTunes. but that did not help.

 

Scripting iTunes to modify Location

I started to code an apple skript to set the Location, but I was not able to read the Location attribute of the track item. All I got was "Missing"-Value.

 

 

 

So I'm stuck at the moment. The austrian-apple-support guy was friendly but his help was to suggest to address the problem to the support community.

 

I think nas-boxes with huge audio librarys are quite common these days so I hope someone has solved that problem.

 

Many thanks in advance.


Mac mini (Late 2012), iOS 6.0.1
  • 1. Re: Transfer nas-based iTunes library from Windows-PC to mac
    Zwockel Level 1 Level 1 (0 points)

    I solved the problem by myself. It took some investigation, coding and debugging. I try to explain what I have done:

     

    First of all some advice:

    1. Don't try this, if you can't code. It's better to ask someone to help you in front of your machine.
    2. Don't assume, that I can help you if something does not work. It's necessary to look at the results and the files and I can't do that.
    3. Try it step by step and of course backup your system and the iTunes-folder at first.

     

    Now to the solution:

     

    The key for the solution is the XML-tag Database ID in the file iTunes Music Library.xml.

     

    Extracting a table with Database ID and path from iTunes Music Library.xml

    I wrote a shell script with AWK and PERL to extract the Database ID and Location Key and saved them into a simple CSV-file.

     

    shell-skript parseiTunesXML.sh:------------------------------------------------------------- ----

    #!/bin/sh

     

    if [ "$1" = "" ] ; then

              echo "parseiTunesXML.sh: scans iTunes-XML-Files for IDs and location of tracks"

    else

       if [ "$1" = "all" ] ;then

              MyInputfile='/Users/<USERNAME>/Music/iTunes/iTunes Music Library.xml'

       else

              MyInputfile=$1

       fi

     

       if [ "$2" = "" ] ; then

     

              awk -f parseiTunesXML.txt "$MyInputfile" 

       else

              MyOutputFile=$2

              echo awk -f parseiTunesXML.txt "$MyInputfile" >$MyOutputFile

     

     

              awk -f parseiTunesXML.txt "$MyInputfile" >$MyOutputFile.tmp

           cat $MyOutputFile.tmp | perl -pe 's/%([0-9a-f]{2})/chr(hex($1))/eig' >$MyOutputFile

           rm $MyOutputFile.tmp

       fi

    fi

    -------------------------------------------------------------------------------- ---------------

    The skipt is envoked in Terminal with

    ./parseiTunesXML.sh all out.csv

     

    It parses the iTunes Music Library.xml in the music folder of the user <USERNAME> (you have to change that, of course) with the AWK-file parseiTunesXML.txt. The location is urlencoded which is decoded through the following perl-command. I encountered some more problems with filenames having an ampersand '&' oder a semicolon ';'. I left these file out an first (I come back to them later).

     

    The following AWK-file does the extraction of the files with one of theOldPath-constants in their location. You need to adapt these constants to your situation. But pay attention these constants are urlencoded-constants. So if you've got some umlauts in your old path you have to use the encoded version of your path. Take a look at your  iTunes Music Library.xml - for example with TextEdit. Then you will see the different versions of the path's and you can easily adapt theOldPath-constants.

     

    The AWK-script does not only the extraction of the OldPath. It exchanges the part of theOldPath with theNewPath-constant. Therefore the content of the out.csv is the should-be state.

     

    awk-file parseiTunesXML.txt:-----------------------------------------------------

    BEGIN {

       locTrackID = 0;

       locName = "undefined";

       locArtist = "undefined";

       locLocation = "unknown";

       OFS = ";";   

       theOldPath1 = "file://localhost//Nas/audio/";

       theOldPath2 = "file://localhost//nas/audio/";

       theOldPath3 = "file://localhost//Nas/doc/";

       theNewPath = "/Volumes/audio/";

    }

     

    END{}

     

    function skipThisFile( path ) {

       found = index(path, "&#38;");                    # ampersand

       found = found + index(path, "%3B");           # semicolon

       return found;

    }

     

    function getXMLValue( name, string ){

       localValue = "";

       leftPos = index( string, "<" name ">" );

       if (leftPos >= 1) {

          rightPos = index( string, "</" name ">" );

          leftPos += length( name ) + 2;

          localValue = substr(string, leftPos, rightPos-leftPos);

       }

       return localValue;

    }

     

    function getInteger( string ){

       localValue = getXMLValue("integer", string);

       return localValue;

    }

     

    function getString( string ){

       localValue = getXMLValue("string", string);

       return localValue;

    }

     

    function getKey( string ){

       localValue = getXMLValue("key", string);

       return localValue;

    }

     

    getKey($0)=="Track ID"           { locTrackID = getInteger($0); }

     

    getKey($0)=="Name"                     { locName = getString($0); }

     

    getKey($0)=="Artist"                     { locArtist = getString($0); }

     

    getKey($0)=="Location"           {

       locString = getString($0);

       hit = gsub(theOldPath3, theNewPath , locString);

       if (hit == 0) {

          hit = gsub(theOldPath2, theNewPath , locString);

       }

       if (hit == 0) {

          hit = gsub(theOldPath1, theNewPath , locString);

       }

       if (hit < 1) {

          next;

       } else {

          ignore = skipThisFile( locString );

          if (ignore > 0) {

             # to speed up the search, we leave the records to ignore in the file

             print locTrackID OFS locString ;

          } else {

             # we do the urldecode via a tiny perl statement

             locLocation = locString;

             print locTrackID OFS locLocation ;

          }

       }

    }

     

    {

     

    }

    -------------------------------------------------------------------------------- ---------------

    Some caveats:

     

    I got the impression, that iTunes changes the Database ID when started. (I don't know why they call it an ID, but what the heck...) So be sure iTunes is running when you use the following apple-Skript to set the location of the files.

    The following AppleScript scans for each file through the whole OUT.CSV until the Database ID is found. This can be very time consuming. I found out, that the Database ID is in the consecutive order in which the file where added to iTunes. So you will speed up the search when you order the track in the iTunes Windows by the Date Added.

    The code in this article is not beautiful, it is not fast and it is of course presented as it is - with no warranties whatsoever. It just worked for me

     

    Setting the location of the files

    So having extracted all the wrong tracks with their Databse ID and their new Location. We can start to change the location of the wrong files. I wrote another script - now using AppleScript - to tell iTunes to walk through its musicfiles (aka tracks), tests their location and set them if they are missing.

     

    The filepath-variable contains the location of the out.csv (I used /Users/<USERNAME>/Documents/workspace/bin/iTunes) and should be adapted.

     

    The script scans all selected files in the first iTunes window. If a path is missing the script searches with the Database ID of the current track in the out.csv for the new location. The it checks if the location is a correct file and sets the attribute of the track.

     

    -------------------------------------------------------------------------------- ---------------

    global unknownTrack -- as Track

     

    tell application "iTunes"

              set error_msg to false

      activate

     

              display dialog "This script will relocate tracks, whose corresponding file is missing" & return & return & ¬

                             "This action cannot be undone." & return & ¬

                             "Search missing tracks in:" buttons {"Selection", "Cancel"} default button 2 with icon 2

              copy the result as list to button_returned

     

              set counter to 0

              set the stored_setting to fixed indexing

              set fixed indexing to true

              set filepath to (((path to documents folder) as text) & "workspace:bin:iTunes:out.csv")

     

              if button_returned as text = "Selection" then

     

                             set these_tracks to the selection of browser window 1

                             if these_tracks is {} then error "No tracks are selected in the front window."

     

                             display dialog "Beginning process." & return & return & ¬

                                            "One moment…" buttons {"•"} default button 1 giving up after 1

     

                             set changeAll to false

           -- this is just the part that moves the tracks =)

                             repeat with this_track in these_tracks

                                            try

                                                           if the location of this_track is missing value then

     

                          set unknownTrack to this_track

                          set theTitle to name of unknownTrack as text

                          set newpath to my getfilepath(database ID of unknownTrack as text, filepath)

     

                          if not changeAll then

                                                                                         display dialog "Shall I change the location of the song '" & theTitle & "' to " & return & return & newpath & "'" buttons {"Yes", "All", "Cancel"} default button 3 with icon 2

                                                                                         copy the result as list to button_returned

     

                               log button_returned as text

                                                                                         if button_returned as text = "All" then

                                                                                                        set changeAll to true

                               end if

                          end if

     

                          if (button_returned as text = "Yes") or changeAll then

                               set HFSLocation to POSIX file newpath as alias

                               -- log HFSLocation as text

                               set location of this_track to HFSLocation

                          end if

                          if button_returned as text = "Cancel" then

                               log "STOPPED"

                               error number -128

                               return

                          end if

                          set counter to counter + 1

                     else

                          log database ID of this_track as text

                          set HFSLocation to location of this_track

                          log HFSLocation as text

                          set posixLocation to POSIX path of HFSLocation

                          log posixLocation as text

                     end if

                on error

                     log oldPath

                     log newpath

                     log HFSLocation as text

     

                end try

                             end repeat

              end if

     

              set fixed indexing to the stored_setting

     

              log "Process complete. " & (counter as string) & " tracks were relocated."

              display dialog "Process complete. " & return & return & ¬

                        (counter as string) & " tracks were relocated." buttons {"OK"} default button 1

    end tell

     

    on getfilepath(trackId, csvfilepath) -- integer, filepath

              set returnvalue to "NotFound"

              try

           set csvData to read file csvfilepath as «class utf8»

           set csvEntries to paragraphs of csvData

           set theAmount to count csvEntries

     

                             set done to false

           set i to 1

           repeat until done

                set {theId, thePath} to parseCsvEntry(csvEntries's item i)

                if (trackId is theId) then

                     set done to true

                     set returnvalue to thePath as text

                end if

                set i to i + 1

                if i > theAmount then

                     set done to true

                end if

           end repeat

     

           return returnvalue

              on error

           log trackId as text

           log csvfilepath as text

           log "ERROR"

           return returnvalue

              end try

    end getfilepath

     

    to parseCsvEntry(csvEntry)

              set AppleScript's text item delimiters to ";"

              set {theId, thePath} to csvEntry's text items

              set AppleScript's text item delimiters to {""}

              return {theId, thePath}

    end parseCsvEntry

     

    -------------------------------------------------------------------------------- ---------------

     

    Some manual work - the remainders

    Now to the files with ampersand (maybe the are other characers I didn't come across): after having migrated most of my iTunes library I modified the AWK-file so that the path's with ampersands are now written into the OUT.CSV.

    Replace

          ignore = skipThisFile( locString );

    with

          ignore = 0;

     

    and edit the resulting file for example with TextEdit and replace all occurences of &#38; with &. The run the Apple Script again.

     

    I did not find a solution for files with semicolon in their filenames. I pointed iTunes manually to their new position. Another posibility would have been to change their filenames and use the "ampersand-method" to set their location. If there are a lot of semicolon-filenames a coder could use the OUT.CSV to build a shell-script with move (mv) commands to rename the files by a script.

     

    A last check

    And to check if everything is ok right now, I wrote another Apple Script which scans the iTunes library if there are still missing files and if so to write their Database ID, the artist, the album and their title into another CSV-file:

     

    -------------------------------------------------------------------------------- ---------------

    global unknownTrack -- as Track

     

    tell application "iTunes"

              set error_msg to false

      activate

     

              display dialog "This script will scan the iTunes Library and dump those entries, whose corresponding file is missing" & return & return & ¬

                        "iTunes should remain unchanged." & return & ¬

                        "Search missing tracks in:" buttons {"Library", "Selection", "Cancel"} default button 3 with icon 2

              copy the result as list to button_returned

     

     

              set counter to 0

              set filepath to (((path to documents folder) as text) & "workspace:bin:iTunes:iTunesScan.csv")

              set theString to "Database ID ; Artist ; Album ; Title" & return

              set theResult to my writeTo(filepath, theString, «class utf8», false)

     

              if button_returned as text = "Library" then

     

                        display dialog "Beginning process." & return & return & ¬

                                  "One moment…" buttons {"•"} default button 1 giving up after 1

     

     

                        set sourcename to name of source 1

                        tell source sourcename

                                  set libname to name of playlist 1

                                  tell playlist libname

                                            repeat with i from the (count of tracks) to 1 by -1

                                                      try

                                                                if the location of track i is missing value then

                                                                          set unknownTrack to track i

     

                                                                          set theTitle to name of unknownTrack as text

                                                                          set theartist to artist of unknownTrack as text

                                                                          set theAlbum to album of unknownTrack as text

                                                                          set theId to database ID of unknownTrack as text

                                       log theTitle & theartist & theAlbum & theId

                                                                          set x to my createCsvEntry(filepath, theId, theartist, theAlbum, theTitle)

     

                                                                          set the counter to the counter + 1

                                                                end if

                                                      end try

                                            end repeat

                                  end tell

                        end tell

     

              else

     

                        set these_tracks to the selection of browser window 1

                        if these_tracks is {} then error "No tracks are selected in the front window."

     

                        display dialog "Beginning process." & return & return & ¬

                                  "One moment…" buttons {"•"} default button 1 giving up after 1

     

                        repeat with this_track in these_tracks

                                  try

                                            if the location of this_track is missing value then

     

                                                      set unknownTrack to this_track

                                                      set theTitle to name of unknownTrack as text

                                                      set theartist to artist of unknownTrack as text

                                                      set theAlbum to album of unknownTrack as text

                                                      set theId to database ID of unknownTrack as text

                             log theTitle & theartist & theAlbum & theId

                                                      set x to my createCsvEntry(filepath, theId, theartist, theAlbum, theTitle)

     

                                                      set the counter to the counter + 1

                                            end if

                                  end try

                        end repeat

              end if

     

     

              log "Process complete. " & (counter as string) & " tracks were identified."

              display dialog "Process complete. " & return & return & ¬

                        (counter as string) & " tracks were identified." buttons {"OK"} default button 1

    end tell

     

     

    on writeTo(targetFile, theData, dataType, apendData)

      -- targetFile is the path to the file you want to write

      -- theData is the data you want in the file.

      -- dataType is the data type of theData and it can be text, list, record etc.

      -- apendData is true to append theData to the end of the current contents of the file or false to overwrite it

              try

                        set targetFile to targetFile as text

                        set openFile to (open for access file targetFile with write permission)

                        if apendData is false then set eof of openFile to 0

             write theData to openFile starting at eof as dataType

             close access openFile

                        return true

              on error

                        try

                     close access file targetFile

                        end try

                        return false

              end try

    end writeTo

     

    to createCsvEntry(filepath, theId, theartist, theAlbum, theTitle)

              set theString to theId & "; \"" & theartist & "\"; \"" & theAlbum & "\"; \"" & theTitle & "\"" & return

              set theResult to writeTo(filepath, theString, «class utf8», true)

              if not theResult then display dialog "There was an error writing the data!"

              return theResult

    end createCsvEntry

     

    -------------------------------------------------------------------------------- ---------------

    You might want to test this last script on some of your tracks before start. But be aware on the amount of data it produces if let it run on your whole iTunes library

     

    Al last, good luck!

  • 2. Re: Transfer nas-based iTunes library from Windows-PC to mac
    JonHands Level 1 Level 1 (0 points)

    Hi Zwockel,

     

    Thanks for your post. I am willing to give your script a try as i am having the same problem with my nas and iTunes. Just wanting a little leg up to get started, when you say envoke the script is that copying your script into a text editor and the editing it so that it matches my conditions? Then running it via the terminal?

     

     

    Thanks

     

    JonHands

  • 3. Re: Transfer nas-based iTunes library from Windows-PC to mac
    Zwockel Level 1 Level 1 (0 points)

    Hi JonHands,

     

    the shell-scripts (files with the extension .sh) are invoked via terminal. The apple scripts are run via AppleScript-Editor. And the awk-script is run through the shell-Script parseiTunesXML.sh

     

    The shell-scripts and the awk-script are copy-and-pasted into a new text editor and saved under their file names. To run a shell script you will have to declare it as executable with the command 'chmod 755 parseiTunesXML.sh' (You will have to execute this command in the terminal, too.)

     

    The best thing to start with is the last script (see "a last check"). This Apple-script changes nothing within your iTunes-library it just creates a list of files which iTunes can't locate…

     

    You might want to change the path and filename of the output of that script. Then you will have to change this code

     

              set filepath to (((path to documents folder) as text) & "workspace:bin:iTunes:iTunesScan.csv")

     

    where as workspace:bin:iTunes is the subfolder (3rd level!) of the documents-folder of current user. So if you change it to

     

              set filepath to (((path to documents folder) as text) & "iTunesScan.csv")

     

    the file iTunesScan.csv is created just in your documents-folder...

     

    Try that, examine the result and comeback to me afterwards. I will then help you with the next steps...

     

    Zwockel

  • 4. Re: Transfer nas-based iTunes library from Windows-PC to mac
    gazzabhoy Level 1 Level 1 (0 points)

    Hi Zwockel, yours I think is the closest to a problem I have had for a few months now so I hope you may be kind enough to help me get access to my music library again... !

     

    I updated my NAS (Netgear Stora) a while ago and since then for some reason I couldn't connect to it again in the same way from my macbook pro, as I had previously done (by selecting the drive on the left of my Finder window and click connect now) and so I have lost the path to all my music on the NAS. I've narrowed the problem down to this issue with the path (netgear must have changed something in the protocol in how it connects to Mac OS but they have been no help?), and how the Mac connects to the NAS.

     

    My itunes music library.xml says the path to my music is Location</key><string>file://localhost/Volumes/garyreid/MyLibrary/MyMusic/My%20 Burned%20CD%20Collection/3%20Colours%20Red/Pure/01%20Pure.mp3

     

    but when I am connected to the NAS the way I can now, the path is directly to /Volumes/MyLibrary - no reference to garyreid which is the NAS username.

     

    So, I either need to sort out the reason for this change in connection between the NAS and Mac or find a way to mass change the path of all things in my library that were on the NAS to exclude the /garyreid/ part? There is music, film, audiobooks and video and podcasts that are stored in the local itunes music folder that I can obviously still access and don't want to change the path of.

     

    I'd be very gratefull for any thoughts/help.

     

    Ta

     

    Gary

  • 5. Re: Transfer nas-based iTunes library from Windows-PC to mac
    Zwockel Level 1 Level 1 (0 points)

    Hi gazzabhoy,

     

    as I understand it your problem is that the update of your NAS changed the automount function. An easy workaround might be

     

    mkdir /Volumes/garyreid >/dev/null

    mkdir /Volumes/garyreid/MyLibrary >/dev/null

    mount -t smbfs //$NAS_USER:$NAS_PASSWORD@$NAS/MyLibrary /Volumes/garyreid/MyLibrary >/dev/null

     

    where as

    $NAS_USER should contain (or be replaced with) your NAS username

    $NAS_PASSWORD should contain (or be replaced with) your NAS password

    $NAS  should contain (or be replaced with) your NAS' network name

     

    another to solve the problem might be to set a symbolic link into your /Volumes folder

    cd /Volumes

    mkdir garyreid

    ln -s /Volumes/garyreid/MyLibrary MyLibrary

     

    the way to change the foldernames in your iTunes-Library is cumbersome and I would suggest to do it only if both suggested possibilities fail.

  • 6. Re: Transfer nas-based iTunes library from Windows-PC to mac
    gazzabhoy Level 1 Level 1 (0 points)

    Hi Zwockel I think that is exactly the problem....

     

    Your first work around seems straightforward but I have no code experience, I presume I just copy and paste this with the correct NAS detail into terminal? Will I have to do this each time to mount the NAS? Also if you have a minute to explain what the two different options are doing that would help...

     

    Thanks again

     

    Gary

  • 7. Re: Transfer nas-based iTunes library from Windows-PC to mac
    Zwockel Level 1 Level 1 (0 points)

    Hi Gary,

     

    you're welcome. Both are shell-commands which are to be executed via Terminal. And both could be just copy-and-pasted into a Terminal-Session.

    The first option opens a new connection to your NAS but opens it under your old folder name:

    /Volumes/garyreid/MyLibrary

     

    The second options sets a symbolic link from the old folder name to the new folder.

     

    I'm not quit sure if the second option is really permanent. (The easiest thing is to just try it.)

    The first one is only active in your current session and has to be executed everytime you logon again.

     

    Hope I could help you.

     

    Zwockel

  • 8. Re: Transfer nas-based iTunes library from Windows-PC to mac
    gazzabhoy Level 1 Level 1 (0 points)

    Hi Zwockel,

     

    getting error message after the second line :

    mkdir /Volumes/garyreid/MyLibrary >/dev/null

     

    Error:"mkdir: /Volumes/garyreid/MyLibrary: Permission denied"

     

    So I created the MyLibrary folder manually in finder and when I run the 3rd line nothing seems to happen other than I get the following from "usage":


    mount -t smbfs //garyreid:xxxxx@STORA/MyLibrary/Volumes/garyreid/MyLibrary >/dev/null

    usage: mount [-dfruvw] [-o options] [-t external_type] special node

           mount [-adfruvw] [-t external_type]

           mount [-dfruvw] special | node


    Ta

     

    Gary

  • 9. Re: Transfer nas-based iTunes library from Windows-PC to mac
    Zwockel Level 1 Level 1 (0 points)

    Which Mac OSX version do you use? The mount-command parameter t selects the file system which is to mount (smbfs = samba filesystem -> typical for "windosw-like" nas's...)

  • 10. Re: Transfer nas-based iTunes library from Windows-PC to mac
    gazzabhoy Level 1 Level 1 (0 points)

    Latest version of OSX, but fear not as I played around with your second option and created a symbolic link called garyreid in the volumes and that seems to work... Can be a bit slow exploring sometimes but overall really happy as I can play my tunes!!

     

    Thanks so much for your help.

     

    Gary