SSH and international characters (crosspost)

Question

Level 1

0 points

SSH and international characters (crosspost)

Hi all!

I know this has been posted before, but I haven't found a working answer and the previous topics are all archived, so let me begin anew. (I am crossposting it from the 10.6 forum.) I have a MacBook with 10.6.4 at home and a Windows 7 laptop at work. What I am trying to do is SSH-ing into my home laptop to perform some simple tasks.
The trouble I am having is no matter what I try, the command line doesn't display Cyrillic characters (I'm Russian, so I kinda need them from time to time). Instead I get question marks in the terminal and codes instead of letters when I type.
I have up till now found two links regarding the topic, one english ( http://www.macosxhints.com/article.php?story=20090712151742269) and one russian ( http://theapplegeek.ru/archives/2761). The latter tells to edit two files: ~/.ssh/environment and /etc/sshd_config, adding LC_CTYPE=UTF-8 to the first one and AcceptEnv LANG LC_* to the second. I tried that and even tried adding PermitUserEnvironment yes to the /etc/sshd_config to no avail. The other link seems inapplicable, as it seems to assume that I am connecting from a *nix machine.
What is there left for me to do in this case?

PS Yeah, I am using PuTTY on the Windows machine.

MacBook, Mac OS X (10.6.4)

Posted on Jul 8, 2010 4:09 AM

Reply

Answer 1

Jul 26, 2010 5:29 PM in response to AntonZoo

AntonZoo, try running the following two commands:


echo ${LANG?unset}
echo ${LC_CTYPE?unset}

Do they return the same value, or do they return different values? Do either of them respectively return


-bash: LANG: unset
-bash: LC_CTYPE: unset

(presuming that your default shell is bash)? The display of two question marks for Я implies that the shell’s locale does not support UTF-8 characters.

If both values are unset, try the following commands:


echo "LANG: " $(LANG='ru_RU.UTF-8' ls -l ?)
echo "LC_CTYPE:" $(LCCTYPE='ruRU.UTF-8' ls -l ?)

They should respectively produce output something like


LANG: -rw-r--r-- 1 AntonZoo wheel 0 26 июл 20:03 Я
LC_CTYPE: -rw-r--r-- 1 AntonZoo wheel 0 Jul 26 20:03 Я

If the setting of each environment variable in turn allows the Я to be displayed, then your .profile file should export the appropriate variable, depending on which result is preferable — either:


# ensure system output is in Russian wherever possible
# with display of UTF-8 characters
LANG='ru_RU.UTF-8' export LANG

or


# ensure system output displays UTF-8 characters
# with recognition of Russian character classification
LCCTYPE='ruRU.UTF-8' export LC_CTYPE

Only one of them is needed — whichever one meets your needs best.

Reply

Answer 2

etresoft

Level 9

57,902 points

Jul 11, 2010 1:59 PM in response to AntonZoo

I think it would be simpler to just copy all the pertinent .dotfiles from the new account into the old account. I don't know how your account is setup, so I don't know which dotfiles those might be. You can always just move them all aside and let it create new ones the next time you connect. Then put them back one by one until it breaks again. Then you've found the offending file.

Reply

Answer 3

Jul 8, 2010 5:11 AM in response to AntonZoo

Make sure that your Windows SSH client is configured to handle UTF-8 (for example, if you are using PuTTY, the settings is under 'File >> Settings' and then 'Window >> Translation'). It should, by default, use some Windows codepage that isn't UTF-8.

On the Mac side, what you did is OK, though it's just plain easier to add it to ~/.bashrc. Albeit, the locale shouldn't be 'UTF-8', it should be something like 'en_US.UTF-8' or 'ru_RU.UTF-8' since the locale is the language followed by an optional '.' and encoding value.

Reply

Answer 4

AntonZoo Author

Level 1

0 points

Jul 8, 2010 6:33 AM in response to J D McIninch

Yeah, I tried changing the translation to UTF-8 and I've corrected the LC_CTYPE to ru_RU.UTF-8 (found the correct name by using locale -a). Still cannot see Cyrillic, though now I can type without problems. This solves the problem partially, as the names actually get completed after I press Tab.

Could the problem be with the commands used? Perhaps ls cannot display international characters correctly? And if so, is there a way to fix that?

PS: Furthermore, I think the problem is with the command, as if I cd into a directory with a Cyrillic name and type pwd, in response I get a correct path with Cyrillic characters.

Message was edited by: AntonZoo

Reply

Answer 5

etresoft

Level 9

57,902 points

Jul 8, 2010 7:36 AM in response to AntonZoo

AntonZoo wrote:
Yeah, I tried changing the translation to UTF-8 and I've corrected the LC_CTYPE to ru_RU.UTF-8 (found the correct name by using locale -a). Still cannot see Cyrillic, though now I can type without problems. This solves the problem partially, as the names actually get completed after I press Tab.

Maybe undo any changes you have made so far and try again. I set PuTTY to use UTF-8 and connected and couldn't see a the directory I just created named "Са́нкт-Петербу́р". But as soon as I did "export LC TYPE=ruRU.UTF-8", it showed up perfectly. I have no idea how to type Cyrillic in Windows. I can't even copy and paste it from a web site like I can do on a Mac.

Reply

Answer 6

AntonZoo Author

Level 1

0 points

Jul 9, 2010 12:18 AM in response to etresoft

Nope, nothing. Could it be a problem with the font selected in PuTTY? I tried redoing everything from scratch, as well as typing the export command - before and after I reedited the environment and sshd_config files. Still "????" instead of a name when I do ls.

Say, were you using ls to see if that directory was there? Because I am really starting to think the problem is with it. As before, pwd shows the current directory correctly and the path in the prompt contains proper characters without any missing out. Plus, I can type in Russian and the autocomplete feature on Tab works correctly.

Reply

Answer 7

AntonZoo Author

Level 1

0 points

Jul 9, 2010 4:46 AM in response to AntonZoo

Ok, so as far as I see ls is not UTF-8 compliant. I've googled the topic and found these posts:
http://lists.centos.org/pipermail/centos/2008-February/052250.html
http://lists.freebsd.org/pipermail/freebsd-i18n/2003-November/000063.html
The first one is about CentOS, the second one - about FreeBSD, but nevertheless. I tried ls | less, ls | more and ls | car - all of them returned 100% correct directory and file names.

So, the final question would be - is there a way to make ls understand UTF-8?

Reply

Answer 8

etresoft

Level 9

57,902 points

Jul 9, 2010 10:41 AM in response to AntonZoo

AntonZoo wrote:
Nope, nothing.

Did you try that command exactly as I wrote? Because I screwed it up. It should be "export LC CTYPE=ruRU.UTF-8"

Could it be a problem with the font selected in PuTTY?

I only use Windows for browser testing. I actually installed PuTTY just to test this. Maybe create a new user account on your Mac and ssh into that. If you have any shell or environment wackiness, that should eliminate it.

Say, were you using ls to see if that directory was there?

Yes.

As before, pwd shows the current directory correctly and the path in the prompt contains proper characters without any missing out.

Same here.

Plus, I can type in Russian and the autocomplete feature on Tab works correctly.

That is some progress. My autocomplete does NOT work properly. I get "cd \320\320\312 yada yada yada". You do have something extra there. It sounds like your bash is working better than mine. But perhaps bash and ls are fighting over who is going to do the encoding and wind up screwing it up.

Reply

Answer 9

AntonZoo Author

Level 1

0 points

Jul 11, 2010 1:51 PM in response to etresoft

Thanks for your help! Still not finished with this problem, but sure making huge progress.

I've tried to create another account and it worked! Do you think there's a way to try and correct this problem with ls or is it simpler to just create another account and transfer all files to that account?

Reply

Answer 10

Jul 14, 2010 7:23 PM in response to AntonZoo

AntonZoo, do you have a .inputrc file in the home directory of the account in which you run ls? My .inputrc file contains the following:


set meta-flag on
set input-meta on
set output-meta on
set convert-meta off
set bell-style visible

Running */bin/stty -istrip pass8 iutf8* in your account’s .profile file might be useful as well, if you know that your Windows terminal program fully supports UTF-8.

Reply

Answer 11

Jul 19, 2010 4:19 PM in response to AntonZoo

AntonZoo, when you run the command *echo -e '\0320\0257'* through your Windows terminal emulator, does Я appear?

If so, try running the commands *touch $(echo -e '\0320\0257')* and *ls -l ?* in succession; is a file named Я listed among the one-character filenames? If not, is some strange character sequence shown instead of Я?

Reply

Answer 12

AntonZoo Author

Level 1

0 points

Jul 26, 2010 11:47 PM in response to Baumkartoffel

Thanks a lot, that helped!

When I ran


echo ${LANG?unset}
echo ${LC_CTYPE?unset}

, the first command returned the expected unset result, while the latter displayed

ru_RU.UTF-8

When I afterwards tried to type


echo "LANG: " $(LANG='ru_RU.UTF-8' ls -l ?)

, it returned


ls: невозможно получить доступ к ?: No such file or directory
LANG:

I have then run

LANG='ru_RU.UTF-8' export LANG

, assuming that LC_CTYPE has anyway returned the needed value. ls now displays Russian correctly.

I added the command to ~/.ssh/environment, so that it stays this way. 🙂

Reply

Answer 13

Aug 6, 2010 4:55 PM in response to AntonZoo

AntonZoo, I should have clarified in my previous note that the expected output from the *ls -l ?* command was based upon the Я file still existing from an earlier execution of the *touch $(echo -e '\0320\0257')* command.

The effect of the LANG environment variable, when set and exported, is the same as if all of the locale-related LC_… environment variables had been set and exported. Thus, if your ~/.ssh/environment file has both LANG and LC_CTYPE set to the same value, you can safely remove LC_CTYPE with no change in behavior.

With LANG set and exported, does the ls command now display Russian filenames properly for you?

Reply