Mail strips headers when forwarding with rules

In the office we let Mail fetch the emails and with rules we 'forward as original' the messages to several accounts on a MSeXchange server. In that process Mail strips a very important header: 'Content-Transfer-Encoding'. No matter what the original encoding is, Mail strips it.

And not only when forwarding to MSX, Mail also strips the header when forwarding to an external account outside the office. If I fetch the email from that external account with Mail, the mail is not readable, because the 'Content-Transfer-Encoding'-header is stripped.

Any ideas? Is it a bug?

Message was edited by: Capono
Forgot to mention: when opening the email in Mail it's readable. Then when 'forwarding as original' Mail puts it's own header encoding 'print quotable' in it.

G4 450, Mac OS X (10.4.8), Mail 2.1

Posted on Oct 26, 2006 1:41 AM

Reply
12 replies

Oct 26, 2006 8:39 AM in response to Capono

I'm not quite sure what you mean by "forwarding as original" -- could you explain exactly what you are doing & how you do it when you forward some message?

Anyway, note that changing the Content-Transfer-Encoding value is not the same as removing it altogether, & that each part of a multipart message will usually have its own Content-Transfer-Encoding declaration along with its individual Content-Type declaration, so you should make sure you are looking at the appropriate sub-header value(s) in what is sent.

Also, "Content-Transfer-Encoding: quoted-printable" is normal for "Content-Type: text/html" parts -- any recent email client should be able to handle it without problems, although older versions of some Microsoft client apps do not.

IOW, we need more detailed info to go further with this, particularly exactly what headers or sub-headers are being changed or removed.

Oct 29, 2006 11:59 PM in response to R C-R

(Strange, I posted a reply last Friday, but it's not here anymore...).

In Mail you can 'forward as original' by two ways: by opening a message and choose menu 'message>forward as original', or by setting a rule to do the same thing.

If I receive a message a let a rule forward it as original to the right account on the exchangeserver, Mail removes some headers. The most important one, the Content-Transfer-Encoding is also removed. In case a message is encoded base64, Outlook can't deal with it.

If I do the forward myself by first open the message and then choose forward as original Mail changes the Content-Type-Enconding to Quoted/Printable.

Mac OS X (10.4.8)

Oct 30, 2006 6:54 AM in response to Capono

From Mail's Help:

"When you forward email, the recipient sees your name in the From column and the date you forwarded it in the Date & Time column.
When you redirect an email message, the recipient sees the original sender's name in the From column and the time when they composed the message in the Date & Time column."


"Redirect" comes closest to forwarding the original message -- but even so, most servers will add additional headers (like new "Received:" lines) & Mail will add headers for "Resent-From," "Resent-Date," & "Resent-Message-Id" to indicate the most recent source, so there really is no way to send an exact copy of the original, regardless of the email client used. You mention "the right account" -- does this mean you are trying to reroute (redirect) a message incorrectly addressed to you to the correct recipient?

If I receive a message a let a rule forward
it as original to the right account on the
exchangeserver, Mail removes some headers.


How do you know this is what's happening? Can you compare the original message to the new one directly? What headers are removed? Look at the "long headers" of the two. Are one or more headers being changed rather than removed? If so, what are they?

The most important one, the
Content-Transfer-Encoding is also removed.


Actually, a Content-Transfer-Encoding header may or may not appear in the header section, even if "long headers" are visible. For a multipart message, you will usually see a "Content-Type: multipart/" header with a boundary value instead. In the actual raw source, you will find each part bracketed by the boundary value, & the Content-Transfer-Encoding of that part in a sub-header, along with its Content-Type. If the original message is multipart, or is changed to that by any additions (for instance, adding text above the "Forwarded message" line in a regular forward), it may only appear that the encoding value is missing, since it may be "pushed" into the various parts. This is in accordance with MIME standards -- if Mail did not do this, it would cause problems with most email client apps.

In case a message is encoded base64,
Outlook can't deal with it.


Does this mean your resent messages are changing the encoding header value to base64 or removing it? Please supply a clearer example of what is happening here.

Anyway, Outlook should be able to handle base64 encoding -- otherwise, it would not be able to accept a common variety of attachments. It is more likely that Outlook is choking on some other MIME declaration, since this is a known problem with older versions of this client application that was corrected in more recent ones.

If I do the forward myself by first open the message
and then choose forward as original Mail changes
the Content-Type-Enconding to Quoted/Printable.


Typically, this would happen with a regular forward, & also should present no problem with a fully MIME-compliant application.

Before deciding there is a problem with Mail itself, it will help to establish exactly what headers are being changed, removed, or otherwise altered; & if the problem occurs regardless of account, server, or message content or type. It is possible the problem is elsewhere, for instance in the server or email client of the recipient. In particular, an Exchange server may process messages with Microsoft's proprietary Remote Procedure Call (RPC) protocol if not properly configured for IMAP access. This might mangle your messages after they leave your Mac.

Message was edited by: R C-R (removed comments referring to "forward as original" since it has been established that it is the same as "Redirect")

Oct 30, 2006 12:58 PM in response to R C-R

The Mac fetches mail for 8 domains, strips
all the garbage and redirects it with rules
to the proper internal accounts, which are
all exchange users.


Can you explain this more fully? What do you mean by "8 domains" & "proper internal accounts"? Are you running your own server? Hosting a domain? If so, is it connected to the other users through the Internet or a local network? What do you mean by "exchange users"? Are they running Mail.app with an exchange account or something else? Who runs the Exchange server?

What do you mean by "strips all the garbage"? Is this something you have set up in a rule, or an observation of the results you see?

The "garbage" in the example original message is the message, which is Content-Type: text/plain; charset="utf-8" -- IOW, it is (or should be) plain text using the utf-8 character set, which doesn't require base64 encoding. In the example message redirected with a rule, the same Content-Type declaration is present & should suffice -- it preserves the content of the original, even without any Content-Transfer-Encoding header, since the content is plain text. The presence of the Content-Transfer-Encoding: base64 header in the original is strange -- this declaration is normally not used for plain text but for things that have already been encoded as 7bit values & need to be decoded by the client app.

The final example shows the plain text of the message. Since it can be compactly sent as Content-Type: text/plain; charset=US-ASCII, this is what Mail has done here.

I'm not sure why Mail uses different forms for a rule & manual redirect. It may be that in the manual case, it assumes you may be editing the message, or need to see the text in its most readable form, while with a rule you would not, but that is just a guess.

However, neither of the example redirects break any rules for MIME format or syntax. As rfc 2045 makes clear:

Note that while decoders must produce a single, well-defined output for a valid encoding no such restrictions exist for encoders: Encoding a given sequence of octets to different, equivalent encoded sequences is perfectly legal.

Perhaps if we understood better what you are doing, we could help with a workaround. As it is, there are too many things we don't know about the original message, the servers involved, etc.

Oct 31, 2006 12:30 AM in response to R C-R

Thanks for your patience. I try to explain.

Can you explain this more fully? What do you mean by
"8 domains" & "proper internal accounts"? Are you
running your own server? Hosting a domain? If so, is
it connected to the other users through the Internet
or a local network? What do you mean by "exchange
users"? Are they running Mail.app with an exchange
account or something else? Who runs the
Exchange server?


We have 8 small business units, each with their own domain. they work on pc's with Outlook and an Echange account. They are all in one building, in one local network. To prevent that virus and spam being spread over the network (pc workers often click on anything, no offence)), I've set up a Mac to fetch mail for them through 8 different popconnections. I then filter the virus and spam out and send the mail over the local network to the accounts at the MS Exchangeserver. I've set up rules to do that. I didn't explain this proper, because I am sure that Mail is the problem. But for you to think of a workaround it is usefull info.

If I redirect the message WITH a rule from the Mac with Mail to a different Mac with Mail, and open the message, it looks like this screendump: http://www.menpmedia.nl/hans/screenshot.jpg
On Outlook clients it looks the same.

I also see that the quotes disappeared in my previous post.

Oct 31, 2006 5:40 AM in response to Capono

There are still a lot of unanswered questions about what you are doing & how, but this much seems evident:

You are not just forwarding original messages to their intended addressee, which is what redirect (or forward as original) is intended for. Instead, you appear to be acting as the addressee, receiving the message, somehow processing & potentially modifying it with something other than Mail.app to remove viruses, & sending that form along to the original addressee.

This is a substantially different situation from what you originally presented. If you modify a message before redirecting it (whether within Mail or with an external agent), Mail will very likely change its headers to accommodate the changes. For instance, if you remove an attachment, you may very well be changing the message from multipart to text, which will cause major changes in the MIME declarations. Even if you just add or remove a single character in the body of a plain text message, you are modifying it.

As I mentioned earlier, MIME does not require that a client app encode a message in the same way it decodes it. Not only is a different, equivalent encoding perfectly legal, it is generally considered a benefit if the encoding chosen is the most appropriate, efficient one for the content. For instance, if a message (or part) can be encoded as plain text in 7 bit form (US ASCII), or as some octet stream (like with UTF-8), Mail is allowed to encode it as such, regardless of its original encoding.

Beyond these considerations, we must consider the (as yet not clearly detailed) effects of the mail server(s) on the message. An Exchange server is not necessarily MIME-compliant, since Microsoft has designed them to support proprietary features. Add client plug-ins or external processing of messages to the mix & it is very difficult to say what message in what format will eventually be delivered to the addressee.

Resolving this will most likely require the server administrator's participation & careful comparison of the original to the various sent & received forms of the message, taking into account the various MIME content types a message might contain & the equivalence of different forms, plus what (if anything) has been changed in the body of the original message before retransmission to the server.

The only thing that hints at a bug or problem with Mail itself in this thread so far is in the now removed source for the rule-driven redirect example you provided, since it seems to show Mail failing to include the encoding header for a plain text, utf-8 encoded message. However, since it isn't clear if this would be the case with other content types, or if the original was properly encoded to begin with (why base64 with utf-8?), plus the uncertainty regarding modifications, it is hard to say where the problem really is.

If you would like, I can suggest some tests that might help better clarify what is actually going on, but you should know that I cannot duplicate the stripped encoding header in my own redirect tests, so it is likely Mail itself is not the problem.

Oct 31, 2006 6:20 AM in response to R C-R

You are not just forwarding original messages to
their intended addressee, which is what redirect (or
forward as original) is intended for. Instead, you
appear to be acting as the addressee, receiving the
message, somehow processing & potentially modifying
it with something other than Mail.app to remove
viruses, & sending that form along to the original
addressee.

No. I do not use another application. In Mail I've set up a rule that if the sender is IN the addressbook, the redirect rule applies. if NOT in the addressbook, the message stays in the inbox. I manually check the contents of the mail that stays in the inbox. If trusted, I add the senders address to the addressbook and apply the redirect rule again. A few times a day I check the inbox for messages that have not been processed yet.

Resolving this will most likely require the server
administrator's participation

I ruled out the server allready. I did the Mac to Mac test in both directions. Redirect it on the first Mac, fetch it on the second Mac, and vice versa.

The only thing that hints at a bug or problem with
Mail itself in this thread so far is in the now
removed source for the rule-driven redirect example
you provided, since it seems to show Mail failing to
include the encoding header for a plain text, utf-8
encoded message. However, since it isn't clear if
this would be the case with other content types, or
if the original was properly encoded to begin with
(why base64 with utf-8?), plus the uncertainty
regarding modifications, it is hard to say where the
problem really is.

Mail does it with all types. Being in base64 meens that Outlook can't read it. Other types are readable by Outlook, even if there is no encoding header. This doesn't mean that the problem is in Outlook. If I add the encoding to the mailfile before opening it in Outlook, Outlook shows it fine. It just needs the encoding.

If you would like, I can suggest some tests that
might help better clarify what is actually going on,
but you should know that I cannot duplicate the
stripped encoding header in my own redirect tests, so
it is likely Mail itself is not the problem.

Any help is welcome. I'll do some tests to and from home.

Oct 31, 2006 1:42 PM in response to Capono

No. I do not use another application.


Then I do not understand why you said you remove viruses from emails as part of the redirect process. And, since senders of malware frequently forge these addresses with those stolen from Windows users' address books, I can't imagine how trusting previously trusted senders can be very effective. Perhaps a better approach would be a rule based on the spam headers & similar, server-level malware checks added to messages.

I ruled out the server allready. I did the
Mac to Mac test in both directions.


I do not understand what you mean here either. There is no "Mac to Mac" test possible with Mail, since all redirects (actually, all sent or received messages) inherently involve at least one server as an intermediary. This is why I keep asking for details about what server, network, etc. is involved. Are you operating a server on one of the Macs? If not, how are you transferring messages from one to the other?

If what I'm asking about is unclear, you may want to check the "How Internet e-mail works" section of E-mail - Wikipedia, particularly the comments about the many alternative possibilities and complications to the e-mail system like using a corporate e-mail system like Exchange add to the mix.

Mail does it with all types.


Are you aware that "all types" includes a large number of possible content types (including multipart & its subtypes) & at least five encoding declarations, some of which are informative, others requiring action on the part of the client, plus a number of content disposition declarations, some of which alter the form of the content? I have been studying MIME & how Mail uses it for several months & still feel like I have only begun to test all the possible permutations. How many have you tried?

Being in base64 meens that Outlook can't read it.


I don't understand why you say this either. The example original message you initially posted included a base64 Content-Transfer-Encoding header value, & your complaint was that Mail removed this header. In any event, any modern email client can handle base64 encoding. What leads you to believe Outlook can't, or do you means something else? If that, can you be more descriptive about the specific behavior of Outlook?

Any help is welcome. I'll do some tests to and from home.


Unless there is much more unsaid I'm not understanding, I suggest you rethink ruling out anything at this point. The problem may not be one single thing but how the various items (servers, clients, message components) interact with each other. In particular, do not assume that anything involving a Microsoft product is fully inter-operative with anything that isn't, since MS has a habit of favoring its own, de facto standards that are sometimes intentionally at cross-purposes with those recognized & maintained by independent standards organizations.

I suggest you start with a set of test messages from an outside source that are known to be problem-free across platforms, & observe how the client-server-client process changes them. In Mail, you can compare the raw source of a sent message (stored locally when the message is sent) to that returned after a round trip through various servers, & to what is delivered directly to the clients. Anything less than this probably won't show much definitive.

Nov 2, 2006 2:43 AM in response to R C-R

Then I do not understand why you said you remove
viruses from emails as part of the redirect process.

Thanks for the advise. Since you asked for it, I gave you the info. But it's not part of the current problem.


I do not understand what you mean here either. There
is no "Mac to Mac" test possible with Mail, ...

Sure, you're right. I obviously ment mac > ISP's server(s) > mac.

check the "How Internet e-mail works" section of E-mail -
Wikipedia
, particularly the comments about the
many alternative possibilities and complications to
the e-mail system like using a corporate e-mail
system like Exchange add to the mix.

Read it. Good info, but doesn't solf the problem (yet).

Are you aware that "all types" includes a large
number of possible content types (including
...How many have you tried?

About 5 different types. It's not the value of the Content-Transfer-Encoding, it's the header itself that gets removed.

that, can you be more descriptive about the specific
behavior of Outlook?

I can't be more descriptive. The behaviour also occurs in other mailclients (Mail, Thunderbird, Outlook Express, Webmail). The Content-Transfer-Encoding header is removed. And there was no local network involved. Tested it at home with one mac, sending it from one account at one ISP/domain, to another account with a different ISP/domain.

I suggest you start with a set of test messages from
an outside source that are known to be problem-free
across platforms, & observe how the
client-server-client process changes them.

I'll do that and get back to it next week.

Nov 2, 2006 9:26 AM in response to Capono

Since you asked for it, I gave you the info.
But it's not part of the current problem.


What I meant was that if you change the message before resending it (by whatever means & by any amount whatsoever), you are no longer redirecting the original message & it is quite likely it will be re-encoded before being resent.

I obviously ment mac > ISP's server(s) > mac.


Is it also obvious to you that because of this, you cannot necessarily rule out the server(s) as a contributing factor? Exchange servers are most suspect, but some "regular" providers do alter content, sometimes by appending text (typical of free account providers), or by removing suspect attachments (anti-virus or parental control efforts). A good test for this is to send yourself a multipart message with attachments & compare the local copy in 'sent' with the received form.

It's not the value of the Content-Transfer-Encoding,
it's the header itself that gets removed.


Unfortunately, Content-Type & Content-Transfer-Encoding cannot be considered independently. As I mentioned earlier, some content can be transmitted perfectly well with 7bit encoding, which (because it is the default) does not require a Content-Transfer-Encoding header at all. Multipart messages are generally sent as 7bit at the top level, with Content-Transfer-Encoding declared individually within the sub-headers for each part, so here again there is no requirement for this header at the top (long header) level. This can make it appear that the Content-Transfer-Encoding header has been stripped when in fact it has been moved to a less visible part of the message.

The behaviour also occurs in other mailclients


Unless I'm misunderstanding something, this is different or new information from what has been provided in previous posts. I'm beginning to suspect the problem is in the original messages and/or in your method(s) of resending them. It isn't impossible that messages are formed with incorrect MIME syntax, which would cause unpredictable results when resent by an email client app that tried to correct it. (This is why I've mentioned the unusualness of the original example you posted.)

Overall, I hope this helps you understand the complexities involved, & why extensive & careful testing may be required to understand exactly what is going on. It may be, particularly in light of this latest information, that there is nothing wrong, that Mail and other client apps are behaving exactly as they should.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Mail strips headers when forwarding with rules

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.