Lost mail in OS Catalina upgrade

Several people have reported lost or blank e-mail messages after updating macOS to Catalina. I lost 50,000 messages. Here's my solution, which requires a back-up of messages made before the Catalina update.

1. Restore the folder V6 from User/Library/Mail to the desktop.
2. Open Mail if it is not already open.
3. Create a new mailbox where you will put recovered messages so that you can review them before storing them with other messages.
4. Select the command "File" - "Import Mailboxes  … ", then select "Apple Mail" as the format.
5. Look in the folder V6 on your desktop for a mailbox that you want to restore, select it, then click "Continue". The import will start. A message will tell you when the import is done.
6. After the messages have been imported, the messages will be in a new mailbox called "Import". Within that folder, they will be either in the simple folder structure that you would expect or in a multi-level folder hierarchy. If they are in a simple folder structure, click on a message in the message list, press command-A to select all the messages, and drag them to the folder that you created in step 3. If they are in a multi-level hierarchy, option-click on the triangle next to the folder called "Import" to expand the hierarchy. Command-click on all folders named "Messages". Click on one of the messages in the message list. Press command-A to select all messages in the message list. Click and drag the messages to the new mailbox that you created in step 3.
7. Progress in moving the messages will be shown at the bottom of the folder list. After all the messages have been moved, delete the "Import" folder by right-clicking on it and selecting "Delete Mailbox  … "
8. Repeat steps 3 to 7 for other mailboxes.

Methods that did not work for me:

• Re-building mailboxes.
• Deleting envelope files in folder V7 and forcing Mail to re-index V7 messages.
• Temporarily removing folder V7 and forcing Mail to re-convert messages from folder V6.
• Restoring a Time Machine back-up from within Mail. Mail crashes.

Oh, it was posted by me as Tripteron. I don’t know why different user names are showing up . Here is my method. Since sharing it, I have had three discussions with Apple, and their engineers have been investigating. I am waiting for their next report.

Mail recovery method

Several people have reported lost or blank e-mail messages after updating macOS to Catalina. I lost 50,000 messages. Here's my solution, which requires a back-up of messages made before the Catalina update.

1. Restore the folder V6 from User/Library/Mail to the desktop.
2. Open Mail if it is not already open.
3. Create a new mailbox where you will put recovered messages so that you can review them before storing them with other messages.
4. Select the command "File" - "Import Mailboxes … ", then select "Apple Mail" as the format.
5. Look in the folder V6 on your desktop for a mailbox that you want to restore, select it, then click "Continue". The import will start. A message will tell you when the import is done.
6. After the messages have been imported, the messages will be in a new mailbox called "Import". Within that folder, they will be either in the simple folder structure that you would expect or in a multi-level folder hierarchy. If they are in a simple folder structure, click on a message in the message list, press command-A to select all the messages, and drag them to the folder that you created in step 3. If they are in a multi-level hierarchy, option-click on the triangle next to the folder called "Import" to expand the hierarchy. Command-click on all folders named "Messages". Click on one of the messages in the message list. Press command-A to select all messages in the message list. Click and drag the messages to the new mailbox that you created in step 3.
7. Progress in moving the messages will be shown at the bottom of the folder list. After all the messages have been moved, delete the "Import" folder by right-clicking on it and selecting "Delete Mailbox … "
8. Repeat steps 3 to 7 for other mailboxes.

Methods that did not work for me:

• Re-building mailboxes.
• Deleting envelope files in folder V7 and forcing Mail to re-index V7 messages.
• Temporarily removing folder V7 and forcing Mail to re-convert messages from folder V6.
• Restoring a Time Machine back-up from within Mail. Mail crashes.

Same, This update has cost me over a day just trying to sort out the damage to my emails it has caused with no clear resolution. First the install was endless then the lost emails. See next response for the details of the problem.

Welcome to the club! Tell us if you find a solution.

I found this in an other discussion thread:

httpshttps://discussions.apple.com/thread/2508

but it was catastrophic, I lost all my pop emails on the iMac of my office. Like for the MBP at home, I downgrade to Mojave.

G'day,

I used the procedure described by triperton to successfully recover 40,000 odd e-mails and attachments. The e-mails were recovered in the original multi-level mailbox structure into which I had filed them. Random sampling has failed to find any damaged e-mails (eg header only, no text).

Even if you have given up and accepted the loss of your e-mails you can still apply the procedure and get them back provided you have a backup of your e-mail system from before the upgrade. This is true even if you wiped your disk to do a clean Catalina install, providing that back up wasn't on the disk you wiped. (A time machine backup wouldn't be).

If you don't have such a backup, and you haven't wiped the disk since the first upgrade to do a clean install, you may still be able to recover your e-mails. For me the Catalina upgrade left the e-mails in the /Users/<your user id>/Library/Mail/V7 folder in folders with names like <hex number>...<hex number>. You may be able to apply triperton's procedure to them. (I haven't tried this but can't see why it wouldn't work.) You would want to apply the procedure to folders whose names are of the form <Hex number>...<Hex number> and whose date is the day of your upgrade. Don't apply it to folders of that form of name whose dates are after the date of the upgrade because these are for your current mail accounts.

I first came across triperton's procedure at https://discussions.apple.com/thread/250970504?page=1 and have added some comments there to amplify the procedure which some may find helpful.

Good luck.

What follows is too much for most people, but here it is in case it helps someone.

When I updated macOS to Catalina, I lost almost 50,000 locally-stored messages and Apple Mail failed to display many others properly.

After recovering my messages from a pre-Catalina back-up (see my method in this thread: https://discussions.apple.com/thread/250716550), I switched to Postbox (www.postbox-inc.com). Since then, I have become frustrated with some aspects of Postbox. I just switched again, this time to eM Client (www.emclient.com).

eM Client does not have the problems of Apple Mail or some of the annoyances of Postbox. It has a nicer user interface than Postbox. It handles Exchange, iCloud, and IMAP accounts well. It seems to get regular maintenance. The vendor's blog says that an update is due in April.

Transferring locally stored mail from Apple Mail to eM Client may not be as easy as you want it to be because eM Client can import only one MBOX folder at a time. There is a neat way around that, but it requires some knowledge. I got Thunderbird and moved all my MBOX folders into the appropriate OS-level folder in Thunderbird. Then I used the Thunderbird-specific import function in eM Client to transfer many folders at once. You can get Thunderbird at https://www.thunderbird.net/en-US/.

Or you could just move all your locally stored message folders to an IMAP account, have eM Client sync with that account, then move the folders to local storage. If you have tens of thousands of messages as I do (120,000 now), then that process will take a long while.

The free version of eM Client allows two e-mail accounts. The pro version, which allows more accounts, is expensive relative to what people are used to paying for good software these days, $49. One can buy a lifetime license for $50.

To use an iCloud account in eM Client, you will need to get an app-specific password from Apple at www.appleid.apple.com and use that as the password.

If you decide to switch back to Apple Mail later, transferring messages in that direction is more difficult. One can use the IMAP method mentioned above. If you don't want to wait days for tens of thousands of messages to transfer, it seems that you'll have to buy another program to convert the eml files exported from eM Client into Apple's format. I tried the demo version of Emailchemy for this step and it worked well (https://weirdkid.com/emailchemy/).

Yup, too much hassle for e-mail functions that should be completely reliable in 2020.

Anyone knows for how long this has been published? If there were only a few hundred users affect I doubt they would mention the problem in their official user guide:

https://support.apple.com/guide/mail/if-emails-arent-displayed-or-look-strange-mlhlp1159/13.0/mac/10.15

Second question: Anyone here how made a clean install of 10.15.4 and restored profiles from 10.14.6 TM backup? Problem still there?

@Tripteron

you say your method (page4 in this thread) works even though you report that over 1500 emails but still missing? Other guys tried it and were not successful. Or do you have an updated step-by-step procedure but I have overlooked?

https://discussions.apple.com/thread/250716550?answerId=251840868022#251840868022

https://discussions.apple.com/thread/250716550?answerId=251849651022#251849651022

https://discussions.apple.com/thread/250716550?answerId=251964281022#251964281022

Apple's "moderator" took out all my information - such as exactly how bad my email problem is.

1. Apple Mail was crashing EVERY SINGLE TIME it opened
2. Many hours and calls with Apple, and latest was to do a full new re-install and restore from time machine
3. The latter has blown up all my mail - I have to 'import' which means losing 10 years' worth of categorization and organization flagging etc., smarboxes, nomenclature, categorization, including information needed for divorce litigation - the reason I told Apple I could not simply delete the accounts and start over - needed my email intact - critical for the legal case I'm involved in - and guess what? SMACKED DOWN.

Contacted Apple Support about the latest problem, and almost lost my mind dealing with their godawful 'callback' that calls me back and when I answer gives a long pause, followed by "This is Apple Support. We tried to reach you. We'll try again later" even though I was THERE on the line trying to talk to them. Eventually I got through, albeit very hangrytired by then...and can not reach the dedicated senior person who promised to stay with me on this case until it was resolved. Got second level support who said 'just reply to the email we sent you'...but there is no email. They can't even generate emails to me to respond to to upload a screen shot, because something is wrong with their system. Some kind of 'lock'.

Got cut off - my bad trying to answer another auto-call from a prior effort to contact Apple'.

Contacted Apple support again and the next 2nd level person was a bit rude 'Well, don't you have a backup?"

Yes...and no...did not have disk space for a full back up, so this is a problem.

The long and the short.

Catalina blew up my Apple Mail and left me USER DOWN since November with Mail.

Efforts to fix the problem have made it worse. The workaround makes it worse. (import vs restore means losing all your organization etc.)

Support system is broken.

Wasted time, lost data and potentially I will be screwed in litigation because I can't access my data.

[Edited by Moderator]

coxorange: yes, those are the folders (and mailboxes) that I mean.

You would not need a new account for each folder, just one IMAP account, which you probably already have. You must also have enough storage in your IMAP account for all your messages.

In your place, I might do these things:

1. Back-up everything.
2. Check message counts in every folder and write down the counts. Add up the counts.
3. Ensure that I had a good back-up of the V6 folder. Put it on my desktop. Lock it.
4. Ensure that I had a good back-up of the V7 folder. Put it on my desktop. Lock it.
5. Copy all the folders, mailboxes, and messages that are in my "On My Mac" account into an active IMAP account.
6. Upgrade to Catalina. Open Mail. Let the automatic import process run. Note how many messages it says it's importing. Compare it with the number that I came up with in Step 2 above.
7. Check message counts in every folder and write down the counts.
8. If all the messages are in my "On My Mac" account, check that they are being displayed properly with bodies and correct dates. If everything is good, delete the duplicate folders and mailboxes in my IMAP account. I'm done.
9. If not all the messages are in my "On My Mac" account or if they are not being displayed properly with bodies and correct dates, see if all the messages are in my IMAP account, displayed properly with bodies and correct dates. If everything is good in the IMAP account, delete the duplicate mailboxes and folders in my "On My Mac" account. Copy all the folders, mailboxes, and messages that used to be in my "On My Mac" account from my IMAP account back into my "On My Mac" account. Check message counts in every folder, display of bodies, and dates. If everything looks good, delete those folders, mailboxes, and messages from the IMAP account. I'm done.
10. If not everything looks good, I have to decide if I want to revert to a previous version of macOS or stay with Catalina and restore messages from the V6 folder using the import command from within Mail (see my earlier instructions for that). My choice is to stay with Catalina but use a different mail program. You may have a different preference.

Tripteron, Many thanks for the list you wrote, which might also help others who face that frightening Catalina upgrade.

re 2b. I would add: Make a list of all mails which are marked unread in case they lose their mark. Even better make screenshots of all mails in all mailboxes/folders (a bit time-consuming, but maybe helpful in case all get mixed up).

re 4. Of course I don't have the V7 folder yet.

re 5. How and whereto exactly? Into that IMAP account's Inbox? Why are you sure one IMAP account (I have an iCloud account I rarely use) would be enough? Wouldn't that throw all my "On My Mac" mails together – and later I would need to sort them through/out again (mission impossible!)? Or not – would the folders stay separated? If yes, why/how?

re 7. I would only have to count the messages in the Inbox (= displaying the sum of the inboxes of all my active accounts – I always empty my Sent box, because I rather keep the sent messages – Cc:ed to myself – in my Inbox) and that IMAP account (Inbox?).

re 8. In "On My Mac" there should be about 32K mails. I need 1.25s to check a message, so for 50K emails it would take about 11 hours*. A nuisance, but not impossible. But how can I/you be sure there are no problems in my Inbox too (around 45K additional messages)?

• * EDIT: Forgot, I would have to check the dates too, so 1.25s wouldn't be enough. BTW How should I check/compare all the dates...?

You can restore the mail from your last backup prior to upgrading to Catalina. All of the mail files will be in your User/Home/Library/Mail/V7 folder.

You can restore that one folder from your backup or the entire Mail folder.

> You can restore the mail from your last backup prior to upgrading to Catalina. All of the mail files will be in your User/Home/Library/Mail/V7 folder.

Not sure what you mean. I'm using Sierra and want to upgrade to Catalina.

I have all my 44,000 emails in Mail – no problem.

So why should I restore before upgrading? As a preparation...?

My Mail folder looks like this:

vbat wrote:

when you download your e-mails from a POP account, they are erased from four distant POP account server

... only if you set up Mail to do so! I never did:

Only after backing up my Mac, I delete my POP mails via webmail from the server.

Tripteron wrote:

My lost messages (recovered from back-up) were all in the "On My Mac" account.

With "On My Mac" account you mean in the local folders which you had created under "On My Mac", correct?

I have 24 of them. According to your report, could it be a good strategy (before upgrading to Catalina)

to set up a new email account for each of these folders and to move the emails there?

Then delete the "On My Mac" folders, upgrade, and later create them again and move the messages back?