Mail.app- 'Remove Attachments' corrupting mails

Just to note that when I said "made an appropriate tweak to the offending line (changed to Content-Transfer-Encoding: 8bit);" it was because I could see that the Content-Transfer-Encoding header was clearly incorrect in that specific case, by manually interpreting the raw message content in the message part below that header.

My guess is that Mail.app has a bug regarding which message part headers (of which Content-Transfer-Encoding is one) to apply where after removing attachments. So say the message began with parts [1,2,3,4]. Part 3 might have been a base64-encoded attachment. That's supposed to get replaced by the text note saying the file's been manually removed, which is itself a message part (call it N). The new message with removed attachments is supposed to be [1,2,N,4], but perhaps the part headers are written out as [1,3,N,4] or a similar variant that causes incorrect headers, and thus corrupted display of part 2's content. This simple hypothetical bug is unlikely to have made it into production, but I suspect something related to it, and I'd not be surprised if the way that message headers change in replies within a thread plays a part. (From memory I *think* I've seen the problem when the attachments are on messages in a threaded discussion, but that could just be coincidence!)

I do not understand a lot of what you are referring to however I have been testing various things with my emails in an effort to find a common thing in the hope that I could find an underlying factor or cause of this awful bug and the only common thing I have noted is that when it happens it occurs only in threads of emails from the second reply onwards BUT not all the time.

And, fwiw, my corruption occurs when I remove attachments from messages that have been sent to me directly from a sender (i.e., not a reply or a forward), and whether or not I've replied or forwarded the message.

Hmm, this is interesting, dme26. As I mentioned, I have only encountered the problem with messages containing earlier replies. Which would generate a series of headers, from what I understand.

But: once I had this problem with a "new" message, without nested replies. This was right after my clean re-installation of Yosemite (to get rid of some bugs like this one), and I had missed to check the option to put attachments at the end of the message. So, the attachment was inserted somewhere in the middle of the text, and wham bang, I got the gibberish text after trying to remove the attachment from the sent message.

This sounds pretty close to your theory of headers getting in the wrong order or something.

That's definitely a useful additional data point, 750 H2C. I knew that the not-at-end attachments were done with multipart encoding, but you may have highlighted how to create a conveniently self-contained, reproducible instance of the problem... (now we just need someone at Apple to engage...)

I just found something out that might shed new light on this problem.

Even very old mail messages with removed attachments are now being displayed in gibberish (Yosemite). I have searched my mail archives for years back, and I keep finding garbled messages all over. And I know for sure that they used to display correctly in the days when they were manually removed.

So, my theory is that Mail has been encoding messages badly for quite some time now, perhaps since forever. It happens when you manually remove attachments in some occasions (whatever they are) but not in all.

However, what I think has changed with Yosemite is that Mail is applying a new, more strict, interpretation strategy of any message encoding (old or new). It probably used to be more forgiving, and had a more intelligent/smart interpretation function that didn't just follow the encoding type but perhaps also looked at the content data and tried to guess the most appropriate format. Now it seems Mail has abandoned that behavior, so all the sudden we start seeing these "chinese" characters everywhere.

Solution (if my theory is correct, of course):

First, Apple needs to stop this bad encoding when manually removing attachements (requires update of Mail). Secondly, it is probably best if Apple reintroduced the smart algorithm they used to have – or a better one (also requires update of Mail).

Let me add more clues from my experience (150 corrupted email so far):

• It only affects email sent/received via Exchange Server.
• About 50% of affected email is virgin email sent by me; not replies or forwards.
• Sometimes after opening Mail.app - and this bit is STUNNING - the affected message displays correctly for a fraction of a second. Within a moment the legible content is rendered as gibberish.

I spent a fair amount of time trying to find an app or plugin that sadly removes attachments from Mail.app - without success. Does anyone have a suggestion?

Agree with your bullet #3.

My stats are different on bullet #2. More than 90% of my affected email was not sent by me, but received from other party.

Not sure about bullet #1.

Bullet #1: not only for mail received via Exchange server for me. My main account is IMAP, and I've seen the problem there. (...although I do have an Exchange account that's active, too.)

Bullet #2: I don't think we have consensus here, although it's likely that only a subset of any of our sets of email use cases is going to trigger the problem.

Bullet #3: indeed I've seen this too, but I'd forgotten completely until you jogged my memory!

Regarding alternative approaches, I've typically just used other MUAs in which this feature isn't broken, e.g. Alpine, Mutt, etc. However that probably doesn't help if you need Exchange support, and it is a faff in any case. However I suspect it would be reasonably practical to trigger from Mail.app a local script that reads the currently selected message, uses a non-broken MIME library to parse it and remove the attachments, and then mail it to you. When I'm less short of time, assuming that Apple haven't fixed the bug in the meantime, I'll post back here if I hack something up. (e.g. I use FastScripts to give me a hot-key to copy message IDs. This is useful to me in various ways, e.g. you can pop up specific messages from the shell by running open 'message:<$MESSAGEID>')

Thanks a whale, dme26!

With this fix I was able to restore rendering of about 98% of affected email. There's a particular knack in identifying the right occurrence of "base64" that needs changing into "8bit" because, as others have pointed out, not all need fixing. In some cases additional fixes rendered the email illegible again.

• 1 occurrence of 'base64": Changing this fixed it in every case;
• 2 occurrences: Most of my email had 2 occurrences and here it was consistently sufficient to fix the 2nd but leave the 1st as is;
• 3 occurrences: These were the most tricky and I didn't find a rule here. Those I couldn't restore belong to this category;
• More than 3 occurrences: I was usually lucky with fixing the 1st occurrence alone.

Having been startled by the corruption of email in Mail.app, I reviewed other email clients. With Exchange Server the options are few:

1. Airmail 2.0 (approx. USD 10): Though pretty and highly functional, much of my archived email was re-timestamped to the date of import, which renders archiving meaningless. And attachments can be deleted only one by one, email by email, which precludes professional use.
2. Outlook for Mac 2011 (Office bundle approx. USD 250): It doesn't read the iCloud address book, which is the most basic thing a Mac client should be able to do. Importing and synchronising options are just glossing over intrinsic lack off features and in turn complicating affairs.

In my case, neither of them is capable of replacing Mail.app in spite of its troubles. And now we have dme26's fix to carry on.

I'm pleased to hear that you were able to fix much of your email. Just to follow up on your testing regarding the occurrences of base64: it's not necessarily correct to change the content transfer encoding to 8bit, although it looked to be the right choice in my case. Some gory details at http://www.w3.org/Protocols/rfc1341/5_Content-Transfer-Encoding.html . Likewise, you only want to change from base64 the headers for content that clearly isn't base64. Soon after the incorrect base64 content transfer encoding occurrences you should see content that *doesn't* look like what's shown at http://en.wikipedia.org/wiki/Base64#Examples (i.e. the regular-length encoded lines starting with "TWFu") when Mail.app has messed up.

In terms of rules for occurrences to change, this will require looking into the message itself in some cases: the part structure can be quite complex, e.g. some parts can be declared to be alternative representations of other parts, and thus Mail.app may not even be showing some of the message parts that you are editing (e.g. HTML email typically carries a text-only equivalent by using alternative parts).

New insight: I see now what Apple does wrong in its "Remove Attachments" command.

When they find a base64-encoded text/html portion in the original message they – for some reason – decide to decode it first. Then, after removing the attachment from the source, they write the decoded text version into the new message but leave the encoding headers intact, ie "Content-Transfer-Encoding: base64".

I verified this by comparing original messages with those created after attachment removals that went berzerk. I haven't been able to verify if this is true for all base64 text/html messages with attachments however. Has anybody found any deviations from this?

Suggested solution (to Apple): Stop decoding any base64 parts of the original message. Just leave them as they are, and write them back into the new message with their original headers.

BTW, this is probably why we notice a momentary flash of correct message text. Mail.app seems to start by spewing out all message text as if it were un-encoded. It then applies any decoding procedures, and when they are done, it updates the corresponding text portions (like a webpage would).

dme26 wrote:

it's not necessarily correct to change the content transfer encoding to 8bit, although it looked to be the right choice in my case.

I just had my first no-go. A message that wouldn't display correctly after changing base64 to 8bit. I tried all permutations of which instances to change, but I couldn't get a proper result. So, since I'm way out of my league here, I tried to change it into quoted-printable just for fun. And that worked just fine.