bash and accents

Not easily. These Unicode characters can be encoded in different ways and still be valid. Apple's HFS+ file system specifically uses the decomposed form, which is probably not what terminal is using. All these tools have powerful abilities to directly use Unicode characters. Play around with them. I don't have time to give specifics now, but I will try to come back later.

I've done a little more research.

Your keyboard generates precomposed Unicode characters. The file system stores everything as decomposed Unicode characters. The version of bash (3.2) that ships with any recent version of OS X can automatically translate between the two in some cases. It will not do that in with wildcard expansion. A few years ago, Bash changed to a GPLv3 license that prevents it from being included by Apple. More recent versions of Bash do slightly better. Apple does include a recent version of zsh that is not quite up to even as capable as Bash 3.2.

However, because HFS+ stores the decomposed version of these characters, you don't have to work that hard. Just do "*c*" and it will find the cédille. The same is true for any accented character.

There is no "encoding" for file or directory names. They are in Unicode-16 decomposed. These names are properly handled throughout the operating system with the exception of Terminal. In the Terminal, all data is just a stream of bytes. The keyboard sends data as Unicode precomposed. There are an infinite number of other possible sources of data and there is no way to identify which parts of those streams are file paths. A few places in the shell can detect when you are working with file or diretory names and can do an automatic conversion for you.

Word 2011 has no problem with such directories. I suggest you just upgrade those systems that are crashing. If one particular 3rd party application isn't using the proper operating system facilities and is trying its own thing, and failing at it, there isn't much you can do. This applies to both Word and various shells. Most shells probably assume file names are UTF-8 or whatever Linux is doing. Real Unicode awareness is rare outside of Apple. You may be able to write a Perl script using Perl's Unicode support and File::Find to traverse a directory. It is difficult to write code blindly. Here is what I think you want:

#!/usr/bin/perl

use strict;

use Unicode::Normalize;

use File::Find;

# Search the current directory.

find(\&wanted, '.');

# Look at each item.

sub wanted

{

# Grab the file name.

my $name = $_;

# Only look at directories.

if(-d $File::Find::name)

{

# Strip out all combining marks.

my $stripped = Unicode::Normalize::NFKD($name);

$stripped =~ s/\p{NonspacingMark}//g;

print "$File::Find::name\n"

if $name ne $stripped;

}

}

This will find all directories in the current directory that have a something that could be causing problems for Office. You could concieveably collect them all and rename them in a script like this.

nicolas michel wrote:

Unfortunately, I never learned perl.

That's fine. You probably don't want to see the more in-depth research I did then 🙂.

But finally, a friend (patpro) found the "grep -P" option.

Then I may have some bad news for you. You must be on Lion. I don't think I ever knew about the "-P" flag in grep - and now it is gone. It seems the -P option is a feature of GNU grep. GNU has been changing it licensing over the years specifically to prevent companies like Apple (or Etresoft) from using GNU software. Apple is switching over to more open and less political licenses and GNU grep was replaced with BSD grep in Mountain Lion.

Nicolas,

this should work in Mt. Lion as well: Use "fgrep"

And this works in Mt. Lion: 😁

bash-3.2$ ls

Hægar Møn frühe frûhstûck testing

Mühe Mœn frúhstùck

bash-3.2$ ls -1 | fgrep æ

Hægar

bash-3.2$

The trick is to use fgrep - literal comparison!

Regards

Léonie

nicolas michel wrote:

fink can install gnu grep, so I don't care about Apple vs GNU fight.

To be clear, these "Apple vs ..." fights are just media inventions. GNU released many popular tools and changed their license in 2007. For years Apple just relied on old versions while it worked on some ideology-free free software. Those tools are now mature and Apple is starting to replace the obsolete, circa-2007 GNU tools with more modern equivalents. It is more like "the World vs Apple" with only Apple playing fair.

But still, your script is really interesting.

Could you please explain theses 2 lines ?

my $stripped = Unicode::Normalize::NFKD($name);

$stripped =~ s/\p{NonspacingMark}//g;

Basically is just removes any combining marks. The next step compares the original version with a version that has any combining marks removed. If they are different, then the original had something special in it.

That was my first attempt and it wasn't very good. A better method is to take any path from the operating system and do:

# Convert the file from UTF8 to Perl.

my $octets = decode('UTF-8', $file);

# And decompose.

my $perlName = NFC($octets);

This will result in a Perl-version of the name that can be used for standard comparisons and matching. It is a long story and very Perl-specific.

In fact what I want to search is quite complex and I wish to start with a quite short list of allowed char,

and then be able to update my script whith chat I'll found on client's disk.

Why not just define a set of allowed characters and check for something not in that set?

Still, I think there is a danger in going too far down a "how do I do x,y, and z" path here. I'm not at all convinced you need to. Windows does have some well-known limitations on allowed file and directory names, but it should fully support Unicode too. Word should not be crashing. A better approach is to identify exactly what is causing Word to crash and under what circumstances. Eliminating any non-ascii characters in a path is a 1973 solution to a problem that should be solved in 2012.