How to completely reset the Messages (iMessage) app in Big Sur, and re-download from iCloud
I recently had an issue where I started from a fresh install of Big Sur on my iMac, and ended up with a corrupt Messages app database. This post is both a description of the problem, and the solution I came up with.
I had Messages running on 3 devices, iMac, iPhone and MacBook, all of which were working fine, syncing messages to iCloud and back to each device. The problem started when I experienced a HD failure on my iMac, and after having it replaced, I decided to start fresh to clear out any legacy junk that tends to accumulate over years of upgrades. I opened Messages and signed in with my Apple ID, then enabled Messages in iCloud. It began syncing, but somewhere along the way, it stalled, and wouldn't load messages past 2014 or so. In order to try to get it going again, I believe I turned iCloud off and back on, although I'm a little foggy on that. I might have also signed out and back in, rebooted, etc. In any case, the download resumed, and it appeared it had worked for a while.
I then started experiencing weird symptoms where my messages would disappear after I wrote them. For example, if I sent an iMessage on my phone, it would appear in the conversation on my Mac, but when I clicked the conversation, the message would briefly appear, then disappear. This happened with several different conversations, and only iMessages were affected. Any SMS messages in my conversations would remain, but iMessages would vanish as soon as I opened the conversation.
I spent several hours on the phone with Apple Support, and they had me go through various steps like taking system reports, screenshots and videos, and uploading them for the engineers to review. The engineers were unable to find a fault, and wanted me to go through further troubleshooting steps like setting up a test user and letting it download my messages from scratch, install profiling software, etc. I don't have time for all that, as this is the device I use for work. All I wanted was to be able to reset the library and start the download fresh. When I asked the support staff how I could just reset the database, they said they didn't have a way to do this, which was surprising. My experience with other apps was that it's normally possible to delete the library and start over.
I did some research, and found several helpful articles to understand where MacOS stores the various library files and preferences. Note that the following were the minimum steps I had to do to get it working, given my configuration. Deleting the Messages folder alone did not work! Many online articles reference files and settings from older versions of MacOS/Messages, but they didn't work for me. I hope this solution helps anyone running Big Sur (and possibly Monterey).
Without further ado, here is what I did:
1) Go to Messages->Preferences->iMessage, turn off "Enable Messages in iCloud", then click "Sign Out".
2) Close the Messages App without signing back in.
3) Delete the following, backing them up if you prefer. (I compress the files or folders, and move the zip files to another location.):
~/Library/Messages
~/Library/Preferences/com.apple.imessage.plist
~/Library/Preferences/com.apple.imessage.bag.plist
4) Empty the Bin! (This is important, because MacOS keeps track of moved files, and failing to delete them permanently will just mean that Messages uses the files that are in the Bin.)
5) Reboot the device! (This is also important, to prevent MacOS from caching the Messages state, etc.)
6) Open Messages and you'll be prompted to login with your Apple ID. Go ahead and do so. You should now see an empty conversation list, and a single "New Message".
7) Go to Messages->Preferences->iMessage, and turn on "Enable Messages in iCloud".
Messages should now begin syncing from iCloud. At first, you may see no messages appear, but give it a chance. After a while, you should see conversations appearing, and once it completes, hopefully all of your messages will be restored from iCloud.
I'm not an Apple engineer, but this seems to be the most logical way to get a fresh start with Messages, and alleviate any corruption. It was strange that Apple Support couldn't guide me to do this. I hope it helps someone else in the same predicament.
iMac Line (2012 and Later)