How to prevent Automator's "run shell script" to create fully decomposed forms of my strings ?

I'm not seeing the problem. I did this:

• created new automator service.
• set it to Service receives text in any application (with Replaces selected text checked).
• added a Run Shell Script action.
• added the text echo été in the action.
• saved the service as Test.
• opened TextWrangler, typed in some garbage text, selected it and ran the service

output in TW was été, not the decomposed form. Is your version of TW up-to-date? Mine is version 3.5.3. is the text encoding set to UTF-8? (check TW's preferences, Text Encoding panel for the default).

well, yeah, if I toss in the pbcopy bit it decomposes. That actually doesn't surprise me: in man pbcopy it says -

* Encoding:

pbcopy and pbpaste use locale environment variables to determine the

encoding to be used for input and output. For example, absent other

locale settings, setting the environment variable LANG=en_US.UTF-8 will

cause pbcopy and pbpaste to use UTF-8 for input and output. If an

encoding cannot be determined from the locale, the standard C encoding

will be used. Use of UTF-8 is recommended. Note that by default the

Terminal application uses the UTF-8 encoding and automatically sets the

appropriate locale environment variable.

standard C encoding is plain text, and would naturally misread unicode.

Otherwise, unicode checker says it's unicode, the text looks correct in TextWrangler the characters are treated as single characters

Ok, the whole &@ thing is new information - where does that come into the picture? and are you passing text into the action from stdin or as arguments? (pull down menu on the left.

did you try following my steps above, exactly, to make a new service, and seeing if the problem recurs? it may be that you and I did something subtly different making the service that's goofing things up for you but not for me.

maybe it would be best if you gave a full example of what you're trying to do.

Alright, I guess I need to give the spiel:

Diagnosing computer problems needs a scientific approach - few things on a compter are actually random, so to discover why things are working incorrectly, one needs to isolate factors. I mean, this is what I see:

• You have an automator service that always produces a particular mistake.
• I have an automator service that supposedly is identical, but never produces that mistake

When we discover what difference between your service/computer and my service/computer - and it might be a very tiny, seemingly insignificant and irrelevant difference - we will most likely be able to fix the problem entirely.

We're both using 10.6.7, and it's doubtful that you've done anything that would modify automator or unix in a significant way. So, what I'm asking is for you to carefully and attentively redo the bullet points I did in first point, so that we can compare results and procedures - I'm looking for some tiny difference between what I did and what you did which explains the differing outcome.

getting second and third opinions is fine, on the off-chance someone has diagnosed this problem previously. But if not, the only way this will be solved is if we sit down together and walk through the steps of what we each did in painful detail to find what we did differently.

well, in terminal echo $LANG results in en_US.us-ascii - that's probably the setting in my local bash shell, which may not be at all the generic setting for other things (that would probably be set in .bashrc or some such, and I don't think those files are run by default in AppleScript or Automator).

from a Run Shell Script action I also get a blank result.

Just to be sure, your Run Shell Script action is using /bin/bash and and input is passed to stdin? the latter shouldn't matter for this, but it's worth checking to be sure.

fascinating. I can't get it to be UTF-8. that's a real head-scratcher; it ought to be set in Language and Text, but I can't see any option for it, and can't find a link to it on the web.

I have a dim memory of specifying ascii somewhere at one point (because I do a lot of scripting, and I didn't want unicode hassles in Terminal) but for the life of me I can't remember where. I'll have to think on that.

just for sureness, do you get the same effect in TextEdit (in both plain text and rich text modes?) I just want to be sure we're looking at a system issue and not a TW issue.

that's the problem that's bugging you? I wish you'd clarified that earlier. yes, I get what you're calling the 'COMBINING' letter in TW and TextEdit, though they both look perfectly appropriate. I suspect that the reason the COMBINING one is being used is that the actual figure is part of the Latin character supplement, with the combining form listed as an equivalent, and the system designers probably thought the combining form would port better between different contexts.

at any rate, you probably can resolve this by using unix' iconv utility to convert the text to an appropriate encoding, so that the grep search doesn't fail. either that, or run your search criteria through unix as well: whichever way gets the two texts into the same encoding.

probably you were clear, I just didn't get it properly for some reason.

you might check out the answer on this page. for the purposes of grep, it's not so much that you need to convert them to a particular encoding; you just need to make sure that they are in the same encoding. that makes the task a little easier.

If you want to do complex or frequent regexping with unicode, then I suggest you switch to AppleScript and get the Satimage osax. satimage is well-designed, and may avoid the problems you're having (which may come from the fact that your process is switching contexts so much - file to automator to shell to clipboard to…). If you can do it all in AppleScript you may sidestep all of that.