Non-US-ASCII filenames in tar, scp, etc.

On the Debian box, setting the locale was all I needed to do. Check out 'man locale' for some more details. Like I mention in my original post, I set $LANG and $LC_ALL, and then a few other $LC_ variables (which in all honesty are probably overridden by $LC_ALL).

I agree, it's strange that it's just the command-line tools that don't seem to like these characters. It makes me think I have something broken in my shell enviroment. I can't find any useful information on the subject, though (I spent most of two hours googling yesterday afternoon).

KEN

Can you clarify? Does it work for you on the original filesystem, or only on the UFS filesystem? Does it work in the finder, from the command-line, or both?

Can you think of anything you changed in the environment, or via System Preferences, or in your ~/.profile, etc. that I might try, or are you confident that it always worked for you out-of-the-box?

Thanks for the reply.

KEN

I get the same behavior from a Terminal with $TERM set to "ansi" as I do in an xterm with $TERM set to "xterm-color". It seems the problem isn't about displaying the filename, but writing it to disk.

Fascinating!

I'm not entirely sure how to enter a filename like that (how to compose the characters). If I cut-and-paste from your post, however, I get somewhere. Pasting into a Terminal running kshell (/bin/ksh) doesn't work. Pasting into a Terminal running bash (/bin/bash) does work.

Once I've created the file, I can also create a tar file of the newly touched file and extract it under either bash or kshell on OS X. That's a step in the right direction. It implies that the filesystem does understand how to deal with the special characters.

If I scp the tar file back to Linux and extract it there, however, the result doesn't look right. It comes out like this:

tester_eÌ?uÌ?.txt

It gets more interesting. If I then attempt to scp this file off the Linux box back to OS X, it works! The resulting file is called tester_éü.txt just like the original one!

This is actually beginning to make some sense. This behavior implies that Linux and OS X encode these special characters differently, causing the problem. OS X is reported to have a different Unicode encoding method than other operating systems, sometimes using two bytes where other operating systems use one byte. That seems to match up with what I see.

Now if I could just figure out what to do about it...

KEN

Well, other operating systems can certainly represent some of these characters with one byte (via national codepages, etc.). In fact, I'm fairly sure that Linux is using a single byte in this case, although I can't provide a definitive reference proving that.

In any case, I think you're right. I probably can't do anything about it, short of rewriting tar or scp to do translation. That's rather disappointing, but I guess I'll have to live with it.

KEN

Aha!

You have a two-byte sequence for the ä character:

braeburn.local:/Users/pronovic> tar -zxvf test.tar.gz
L\303\244ndler in_G,_OP.9No.21.ogg

However, my file has a single byte for that character:

braeburn.local:/Users/pronovic> tar -zxvf music.tar.gz
15.L\344ndler in_G,_Op.9,_No.21.ogg
tar: 15.L\344ndler in_G,_Op.9,_No.21.ogg: Cannot open: Invalid argument
tar: Error exit delayed from previous errors

This means that you created the file differently in your SuSE environment than I did in my Debian environment. How did you compose the filename? Can you tell me what you have $LANG and $LC_ALL set to? Perhaps the locale is influencing the way the filename is constructed.

I feel like I'm getting closer. I think that the solution might be to ensure that if I use characters that aren't part of US ASCII, that I need to be sure they're encoded as multi-byte characters, in order to use them on OS X.