Mail.app: utf-8 messages are not displayed properly

Question

Level 1

8 points

Mail.app: utf-8 messages are not displayed properly

Hi,

I have an issue, and I hope someone can help me for where to look for further information.

From emails, which are received over an mailinglist, several characters (for example German Umlauts) are not displayed properly in mail.app. My first idea was that it could be based on the senders. However, I looked into this and the strange thing is, that I can not identify any pattern:

X-Mailer: Microsoft Windows Mail 6.0.6002.18197 - Content-Type: text/plain; charset="UTF-8": ok

X-Mailer: Microsoft Windows Live Mail 15.4.3538.513 - Content-Type: text/plain; charset="UTF-8": ok

X-Mailer: iPad Mail (9B176) - Content-Type: text/plain; charset=utf-8: ok

X-Mailer: iPhone Mail (9B176) - Content-Type: text/plain; charset=utf-8: ok

Content-Type: text/plain; charset=utf-8 - X-Mailer: iPhone Mail (9B179): garbled umlauts

Content-Type: text/plain; charset="utf-8" - X-Mailer: WWW-Mail 6100 (Global Message Exchange): garbled umlauts

Content-Type: text/plain; charset=utf-8 - X-Mailer: iPad Mail (9B176): garbled umlauts

Content-Type: text/plain; charset="UTF-8" - X-Mailer: Microsoft Office Outlook 12.0: garbled umlauts

Interestingly enough, in looking up the above information I made out the pattern that the garbled messages' headers list Content-Type before X-Mailer, but I guess that can not be the reason?

Am I looking in the wrong place?

Any hints/help would be appreciated.

Best Regards

Tom

MacBook Pro, Mac OS X (10.7.3)

Posted on Apr 23, 2012 1:41 AM

Reply

Answer 1

Tom, DE Author

Level 1

8 points

Apr 23, 2012 6:13 AM in response to Tom Gewecke

Sure, I'll try:

To the left ist the rendering in the mail.app. To the right is the corresponding part in the mail's source.

It _should_ read: "Mit freundlichen Grüßen"

This particular message had been sent by an iPad, and the very same message renders fine on the iPhone and Gmail-Webinterface.

I should note that the iPhone does IMAP on gmail, while mail.app POPs the mails (for local archiving and offline research).

Reply

Answer 2

Tom, DE Author

Level 1

8 points

May 1, 2012 10:47 PM in response to Tom Gewecke

Hi Tom,

isn't this related to the dingbat-trick (https://discussions.apple.com/thread/718405?answerId=3480845022#3480845022) ?

As far as I can see, a special char in signature enforces utf-8. Isn't this the same thing happening here, somehow?

Reply

Answer 3

Tom Gewecke

Level 10

122,094 points

May 2, 2012 6:10 AM in response to Tom, DE

Tom, DE wrote:

isn't this related to the dingbat-trick (https://discussions.apple.com/thread/718405?answerId=3480845022#3480845022) ?

As far as I can see, a special char in signature enforces utf-8. Isn't this the same thing happening here, somehow?

In that case adding a special char forces Mail to create the proper charset=utf-8 header so the encoding used to send the message includes the char. This case is rather different, because some added char seems to be causing Mail to ignore the proper charset=utf-8 header when reading the email. Maybe etresoft could figure it out...

Reply

Answer 4

Tom Gewecke

Level 10

122,094 points

Apr 23, 2012 6:27 AM in response to Tom, DE

The source is being misread as Latin-1 instead of UTF-8. For some reason Mail is apparently not recognizing the UTF-8 in the incoming content-type header, but I don't know why this would happen.

Are these multipart messages (e.g. an html copy and a plain text copy) with multiple headers that might have different encodings?

Reply

Answer 5

etresoft

Level 9

56,326 points

Apr 23, 2012 7:48 AM in response to Tom, DE

Tom, DE wrote:

Interestingly enough, in looking up the above information I made out the pattern that the garbled messages' headers list Content-Type before X-Mailer, but I guess that can not be the reason?

Is what you printed really the output from Mail > Save As > Raw Message Source? Because all of them are wrong.

The x-mailer line is a no-op. It is for information only.

The content-type line should be completely on its own line. It could be split into two lines but only if the second line starts with some whitespace.

Reply

Answer 6

Tom, DE Author

Level 1

8 points

Apr 23, 2012 11:56 PM in response to etresoft

Thanks for trying to help me both of you.

@Tom Gewecke: I was not able to find any multipart messages that are rendered false. Therefore I presume that this issue applies only to plain messages.

@etresoft: *blush* No, that was me, trying to avoid copying of sensitive information and to enhance readability by putting them on one line (which I failed because I was not aware of the text width here).

The content types are on two lines mostly. Are there any other operative headers which I could look into?

Reply

Answer 7

etresoft

Level 9

56,326 points

Apr 24, 2012 10:55 AM in response to Tom, DE

e-mail is very hard to debug in a web forum. If you want, you can send one or more copies of the message to me at info at etresoft dot com

Ideally, export the messages as raw source, compress them, and send the compressed archive.

I've been doing e-mail for a long time. I can probably tell you exactly what is causing the problem.

Reply

Answer 8

Tom, DE Author

Level 1

8 points

Apr 30, 2012 4:37 AM in response to etresoft

Thanks again for your effort. However, I was reluctant to forward the full source (possibly including sensitive information) since the mailing-list is member only, and there are terms and conditions, etc...

I was also aware that web forum debugging is kind of difficult, but I was hoping for hints of where to look at.

Anyway, it turns out that I was able to spot the problem in very unimpressive typographical quotation marks, which are used as part of the footer, which is attached to each plain text message (but for some reason not to multipart-messages).

The provider now replaces the culprits with "normal" quotation marks.

Best Regards,

Tom

Reply

Answer 9

Tom Gewecke

Level 10

122,094 points

Apr 30, 2012 10:08 AM in response to Tom, DE

Tom, DE wrote:

Anyway, it turns out that I was able to spot the problem in very unimpressive typographical quotation marks, which are used as part of the footer, which is attached to each plain text message (but for some reason not to multipart-messages).

Thanks much for that info! Very odd, however, I am baffled what mechanism would cause such a footer to override the charset= header in Mail.

Reply

Answer 10

Sep 22, 2012 10:40 AM in response to Tom, DE

I have the same problem with encoding after i upgrade to 10.8.2 (mail.app 6.1). I notice that both my MBP and iMac have the same problem. And i check the mail.app (6.0) out before upgradering and it was just fine.

So something happend between 6.0 and 6.1 thats make the encoding stop working in the automatic setting.

I hope some one find out, or apple them self solve this problem what have hapend.

Reply

Answer 11

Tom Gewecke

Level 10

122,094 points

Apr 23, 2012 6:00 AM in response to Tom, DE

Could you post a screenshot of an example of the garble?

Reply