Mail.app- 'Remove Attachments' corrupting mails

Thanks dme26 for the help on detecting text not encoded with base64. This is key for detecting those instances that I hope to be fixing soon. And I’m hopeful that Rich In Space’s discovery of the underlying issue in Mail.app causing the gibberish might mean we are near a permanent fix. In the interim, I transfer email with attachments to be removed into a local folder to have a controlled environment to do the removing in. I should be optimistic.

But I made a startling discovery today: A local archive of 10k worth of mail, many of which had attachments removed 6 months back without issues (I screened a month ago) suddenly had some 200+ gibberish mails without me touching the folder with ‘Remove attachments’ since. Affected mails are not those fixed a few day back, I had flagged them.

Mail appears to modify my local email on its own. Does anybody have a clue how to prevent mail from doing that?

Mail is apparently becoming a killer app, but not like Apple meant it to. 🙂
Apple must take care of this now. Not only rewrite Mail to get rid of this problem, but also provide some kind of retroactive tool to fix the kind of backlog disease you describe.

Don't worry felix_the_cat, Mail.app is not modifying those old messages at all. They were actually badly formated already when you removed their attachments long ago. Mail.app just used to be more fault tolerant when reading them than it is nowadays. This is actually one of the things that led me to my previous analysis.

I wish I could suggest you just sit tight and wait for Apple to fix this, but the truth is there isn't really any reliable method to detect with 100% certainty if a text is base64 or not. Once you mess the headers up, you're left with guessing …and manual human interpretation.

However, I'm fiddling with a homemade Applescript right now that's been able to "guess" all my mail types correctly this far. But it's based on the cruel assumption that neither me nor any of my friends speak chinese 😉 If I can find the time, I will try hack something together that may relieve the pain until Apple comes to rescue. But please, keep breathing until then.

This problem is also affecting me.

I hope Apple fix this soon, it's a real problem.

Thanks a whale, Rich in Space, for the insight. I will continue fixing my Mail.app email in good faith.

I've fixed another 600 corrupted emails this weekend. And I found time to to review the observations of dme26 and Rich In Space on the message encoding prior and post removal of attachments. Two things continue to amaze me:

• Some 60-80% of my emails have so far allowed the removal of attachments without problems, with 'Quoted printable' or '7bit' encoding. How come Mail.app managed to keep them intact?
• The discussion about the grated-out "Rove attachment" button and purported emails is as old as 2011. And collective wisdom of this discussion group managed to produce a workable rescue fix. How come Apple couldn't with 4 years of time on its hands?

Apologies, I have to re-post my last post to eradicate the typos.

I've fixed another 600 corrupted emails this weekend. And I found time to review the observations of dme26 and Rich In Space on the message encoding prior and post removal of attachments. Two things continue to amaze me:

• Some 60-80% of my emails so far permitted removal of attachments without problems with encodings like 'Quoted printable' or '7bit'. How come Mail.app managed to keep these intact?
• The discussion about the grayed-out "Remove attachment" button and corrupted emails - two issues that I suspect are linked - is as old as 2011. And collective wisdom of this discussion group managed to produce a workable rescue fix besides giving Apple clues about the way forward. How come Apple couldn't address such a grave issue in 4 years time?

Is Apple abandoning Mail.app and its users?

Answering your questions above felix_the_cat:

• The bug probably relates only to encoded (or transcoded) text, eg base64. Note that '7bit', '8bit', and 'binary' mean that no binary-to-text encoding on top of the original encoding was used. So it was quite expected that all those emails you mention were left intact.
• Yeah, I think they are onto this bug and will get there eventually. But to their defence, it's quite a nasty one to figure out.

Normal debugging procedures use versioning tools to step back until a date where the problem didn't occur. They then draw the conclusion that the bug was introduced in some of the code that was changed after that date. In our case, there actually was an "invisible" bug in the code already before the date when the problem started. This is what happens when you have a lot of compensatory code executing in cascade - one fixing the troubles from another. After a while the coders don't know who is doing what and where. And you switch testing paradigm and start testing on a higher level. If everything looks ok at that level then you assume everything is also working as it should below. The API (or whatever) is then signed as passing the tests, but in reality you have burried a ghost deep inside the machine.

To find these bugs you normally need to know both architecture and coding, and have the patience to trace all code back to the roots - "root cause analysis". Now, if they could only read these discussions they would save themselves so much time. But you know, prio's and stuff...

Hi Guys,

I appreciate all this knowlwge but as a person who doesn't really get all the talk of bits and the like is it possible to translate the fix you have all discussed into a something I can understand and this do easily to fix this - step by step sort of?

One thing I have noted in my case after monitoring my own emails for a good while is that only the mails that are replied back to me which still include my original mail signatures are affected.

My signature has two png files in it and I note that when they comeback to me in a reply and they are placed by mail at the bottom of the reply email side by side and out of the original sequence of order then the fault happens once I remove the emails.

I have changed the fromat of my signatures to jpg and giffs over time and this does not change it. I am currently testing this using only one png file.

Here's how:

1. In Mail.app create a local folder (say 'Repair') and move affected email here (drag & drop). You may keep Mail.app open during the next steps or close it, it doesn't matter.
2. Find the folder you created in Finder: 'Users/[your name]/Library/Mail/V2/Mailboxes/Repair.xmbox'. By default, 'Library is hidden to prevent people from messing things up accidentally. You make 'Library' visible by going to your home folder [your name] then click 'Go' in 'Finder' then holding down the 'Option' key. It will appear like magic.
3. Once you put your hands on 'Repair.xmbox', look for email in there. The directory structure can be elaborate. It is important to find each message, named as something like '335458.emlx'. Messages are always in folders called 'Messages' at the bottom of each directory. These need fixing even in case Finder preview may render some or all of them correctly.
4. Open each email with TextEdit.app. In the body, search for instances of 'base 64'. If you are able to read the text underneath at least partially (see screenshots), it means it is wrongly tagged as 'base 64' and you should replace the string 'base64' by '8bit'. After you'r done, save (Cmd+S).
5. Repeat for all email in Repair.xmbox.
6. Once done, verify results by opening Mail.app and the folder 'Repair'. Move fixed email back to its original location. Check if there's any affected email left. If so, it would be isolated now in the directory structure in Finder and you may need to take a second look.
7. If you have lots of affected email, I recommend a shortcuts to quicken the process. Set TextEdit.app as deafult editor (for the time being). Cmd+F to search for 'base64', Cmd+V to paste the string '8bit', Cmd+Q to quite TextEdit. With that I manage close to 10 email per minute.

This is how genuine 'base64' content looks - Do not touch the string 'base64' here:

Non-'base64' content may look like this - This occurrence of 'base64' should be replaced with '8bit':

Hi Felix the cat,

Thank you so much for the time and effort you have taken to help me with that absolutely easy to use set of instructions - greatly appreciated.

Now it is just up to Apple to fix this on a permanent basis and hopefully even look at some of the other rotton things they have done to mail in this incarnation of it.

Regards

Bruce

Two reflections on this:

1. In all my affected messages, there are always two instances of the "base64" string, both of which are followed by something that is clearly not base64-encoded. In some cases, replacing both go these won't do the trick. Only the second one should be replaced. So I have moved on to just replacing the second one, but …
2. … in some other cases, replacing with "8bit" doesn't work. Then I use "quoted-printable" instead, and since that always seems to work, I've discarded "8bit" as my standard remedy.

So: My SOP is to always replace the second "base64" instance with "quoted-printable". It works every time.

A third reflection is that this "base64" circus is more or less regular for me now, when I try to remove attachments from an answer I've written to a thread of messages where there has been at least one earlier attachment (incoming or outgoing, doesn't matter) which I've removed. The strange thing is that my colleague also uses Mail (some earlier version) and when I get copies of some of her correspondence where there are attachments invloved, removing these will NOT induce the problem.

Well, I'll just be d*mned, 750 H2C. My first post on this thread was almost 3 years ago, on a thread that started in August, 2011, and this is the first fix I've seen that works on my corrupted messages - replacing the 2d occurrence of "base64" with "quoted-printable." Great work, to you, dme26, felix the cat!, and several others who have helped us noodle this over the years!

Isn't it grand that we Mac users can now edit the raw text in hidden Library files to be able to read email messages on the Mac platform. Maybe in another 3 years I'll start to forgive Apple for this inexcusable inoperability.

An update: I've now repaired about 50 emails from various senders, over a few days. Most messages have 2 occurrences of "base64" and a few have just 1 occurrence. Replacing the 2d or only one with "quoted-printable" always has worked. A few oddities:

- TextEdit on earlier versions of OS X often finds fewer occurrences of "base64" and often finds no occurrences. We didn't troubleshoot that to death; we could have been executing the search incorrectly (unlikely). Best to stick with TextEdit under Yosemite, or another text editor of your preference.

- there may be no way to search for corrupted emails; in Mail.app, using the black diamond stood on end with the question mark in the middle, we usually turn up a few results, mostly just from odd characters around signatures, but miss most of the truly corrupted emails

- search results on senders of emails where attachments have been removed are bizarre: the email in the Trash folder shows as a result, with the attachment still attached; the near-duplicate email (i.e., the one without the attachment, outside the Trash folder) does NOT appear in the search results

- occasionally, moving an email off iCloud (and maybe other IMAP mail servers) to another folder "On My Mac" generates 2 .emlx files -- we didn't test this extensively, and the workflow may have involved the removal of attachments while the email was in the "On My Mac" folder. We also were able to produce a 3d .emlx file while working with the same one email message. When this happened, we used TextEdit to make the base64 revisions to each .emlx file. Back in Mail.app, even after the message was fixed and moved back onto iCloud, the 2 or 3 .emlx files remained in the "On My Mac" folder. They disappeared once Mail.app was quit and re-launched.

- on one occasion where we think the wrong (1st of 2) base64 occurrence was changed, we lost the entire text of the email and couldn't figure out how to make it reappear (moral: until you're comfortable doing this "under the hood" stuff, you may want to duplicate the .emlx files you're fixing, so that if you experience text loss or other weirdness, you can revert to the copy)

- using a short-cut for "quoted-printable" through TextExpander makes this process MUCH simpler; there's probably a better way to automate this tedious process, but I'll leave it to you coder types to show us that way

HTH,

Ted

I've been using Mail to access remote IMAP email for years and I never encountered this problem -- until I bought a new iMac recently, and suddenly I am encountering this problem (Mail 8.2 on Yosemite 10.10.2). I suppose it could be related to a coincident change in our company's mail server, however.

I'm glad to hear there's a way of fixing the problem. However the problem occurs fairly infrequently for me, and I am not inclined to go editing the affected emails -- especially given that I have a few different accounts on separate, remote IMAP servers. I have found that I can at least view the message body without it being garbling into Chinese and special characters by selecting "View > Message > Raw Source" in Mail.