Encoding problems with Perl and iTunes on macOS

Question

Encoding problems with Perl and iTunes on macOS

Hello everyone,

I am new to the communities, as I rarely have to ask for help. But I have a problem I don't know how to solve with Perl and iTunes on macOS.
My computer have macOS 10.9.5, iTunes 11.4 and Perl 5.16.2, but I had have the same problems with my previous computers.

As a record collector, I manage my collection with a php/mysql app I developed by myself.

When I buy a CD, a vinyl, a tape or files, I usually rip them then add them to iTunes, and add a few other files to the corresponding folders in the iTunes folder structure : a log file (with the information of the CD rip by XLD, or the lineage of the tape or vinyl rip, or where I bought the files), a md5 file, and for the downloaded mp3s, an jpg file of the cover.
My php/mysql app have informations about the existence of these files, so I am able to know if there is something missing for a record or another, like the cover, the kind of encoding of the file, etc.

To make the connection between the files in the iTunes folders and my app, I have a perl script that run every night, and that check the new files I added, so it is able to complete my list automatically. Everything works quite fine except that... I have problems with the different encodings used by macOS and iTunes, when there are accented characters.

An example should help to understand

I have a record by a band called "A Hawk And A Hacksaw" : "A Hawk And A Hacksaw And The Hun Hangár Ensemble". Tracks titles contain a few different accented characters.

As I read on internet, macOS seems to utf8 encoded, so in my perl script, I have these both instructions :

binmode(STDIN, ":utf8");

binmode(STDOUT, ":utf8");

It gives me with one the track of this album the above informations.

From macOS :

filename = ~/Music/iTunes/Music/A Hawk And A Hacksaw And The Hun Hangár Ensemble Test/A Hawk And A Hacksaw And The Hun Hangár Ensemble Test/01 Kiraly Siratás.m4a

From iTunes files, using MP4::Infos :

band = A Hawk And A Hacksaw

album = A Hawk And A Hacksaw And The Hun Hang?r Ensemble

track title = Kiraly Sirat?s

cover = NO COVER

encoding = alac

Which is incorrect.

If I only keep the first line :

binmode(STDIN, ":utf8");

I still have the same problem.
But if I only keep the second line :

binmode(STDOUT, ":utf8");

It gives me the above informations for the same track :

From macOS :

filename = ~/Music/iTunes/Music/A Hawk And A Hacksaw And The Hun HangaÌr Ensemble Test/A Hawk And A Hacksaw And The Hun HangaÌr Ensemble Test/01 Kiraly SirataÌs.m4a

From iTunes file with MP4::Infos :

band = A Hawk And A Hacksaw

album = A Hawk And A Hacksaw And The Hun Hangár Ensemble

track title = Kiraly Siratás

cover = NO COVER

encoding =alac

Which is not better but this time the filename is incorrect, and the infos are correct.
When I deactivate both binmode lines, the problem remains the same as the second example.

I tried a lot of things, with different Perl modules, different methods, but I am still unable to have things correct.

Is there anyone around here that could know what to do to have both the macOS and iTunes infos correct ?
Maybe I am doing the things wrong, but I don't how to do it right.

Thanks for your help.

Pascal

Posted on Mar 3, 2018 2:46 PM

Reply

Answer 1

etresoft

Level 8

46,672 points

Mar 3, 2018 5:42 PM in response to Pascal Berest

Hello Pascal,

First of all, I should warn you that what you are attempting to do is exceeding dangerous. iTunes is not meant to be extended like this. You seem to have avoided any data loss issues because you are still running old versions of OS X and iTunes. If you ever try to upgrade to get a new computer, your carefully constructed database is almost certainly going to go down in flames. While you have it running, I suggest you start looking for ways to export all of your data into some portable format or an external system.

Regarding the actual encoding problem, it is difficult to say. Those binmode directives just specify the expected encoding of standard input and the encoding for standard output, respectively. macOS is not necessarily "utf8 encoded". UTF8 is a standard and ubiquitous encoding, but there are many others and macOS supports them all. Your code needs to properly detect and handle the encoding that it receives. Those directives only apply to standard input and standard output. I don't know how your scripts are getting input and output.

I strongly recommend using UTF8 internally for all of your data and properly specifying UTF8 as the encoding used for any files you create and make available to iTunes. How you do that is dependent on the details of whatever you are doing.

You can't necessarily look at the output either. Where are you getting this output from? There could be other layers of encoding and encoding adjustments that are going on. I think the best approach would be to remove all the binmode directives and see what breaks. That will tell you where you are getting binary input or output that isn't being properly handled.

Finally, I don't know where the original data is coming from. Although UTF8 is standard and ubiquitous, there are a few platforms where it is not commonly used. Unfortunately, one of those is Windows. If you are pulling data like this from the internet, there is a strong possibility that much of it was created on Windows. It may be encoding using any number of Windows "code pages". The actual code page used may or may not be specified. Perl is very good at this and there are extensive Unicode modules designed to try to make sense out of mystery Windows encodings. But it isn't easy.

And that gets me back to my first point. I strongly suggest making sure the data you have is good, correcting it by hand, if necessary. Then start looking for some other system to manage it. Exporting what you have now and importing it into some other system will be plenty of work. Do it now while you still can.

Reply

Answer 2

Mar 3, 2018 4:30 PM in response to Pascal Berest

Any food on this stackoverflow article that gives resolution? Perhaps Chas. Owen's PERL_UNICODE=sdl environment variable suggestion?

Reply

Encoding problems with Perl and iTunes on macOS

Similar questions