I upgraded to Yosemite last Wednesday (October 22) but started experiencing the Force Quit / Out of Memory errors a few days ago (Monday, October 27). After checking Activity Monitor, it was easy to see that it was indeed Mail.app causing this issue, with its memory usage showing 64.19GB. It's been an inconsistent timing of these errors, and, of course, never at the right time.
Anyway, yesterday I tried the steps outlined in the "Guide: How to solve Yosemite memory leaks and CPU usage" post by Luis_Mercado here in Apple Support Communities. These steps included Booting in Safe Mode, Resetting the System Management Controller (SMC), and Resetting the PRAM. I had hopes that Luis' guide would work, but unfortunately, the Force Quit / Out of Memory errors persisted today. While Luis' post didn't correct my issues, it may help others, which is why I am referencing it for others to try.
So with a scheduled call with Apple Support tomorrow morning, I went in search of anything else I may try in the interim. I came across this other thread here in Apple Support Communities, "Re: Yosemite mail memory leak?," that mentioned a potential correlation to those using GoDaddy IMAP accounts. (Subsequently, it was that thread that led me to search for terms "GoDaddy IMAP AND Mail.app memory leak" that led me to this thread.)
It was java_4_me's post in the Re: Yosemite mail memory leak? thread that prompted me to go check my IMAP incoming port settings for my 4 GoDaddy accounts in Mail.app. Turns out that the upgrade to Yosemite must have reverted my port settings back to 143, while leaving the 'Use SSL' checkbox checked, all unbeknownst to me. So I went back and changed the ports to 993 and so far, after 4+ hours, Mail.app has been behaving at around 89-90MB and stable. Far cry from the 64.19GB that it was getting up to earlier today and intermittently since troubles began on Monday. And it's currently longer than any stretch I've been able to go without issue.
My fingers are crossed. I definitely know I was using port 993 on Mavericks. Curious if anyone else with a GoDaddy IMAP account (or any IMAP account) has checked their incoming SSL port settings since the upgrade.
BTW, I have not attempted the suggested fix from this thread about unchecking the 'Store draft messages on the server' checkbox. It has remained check, as it was on Mavericks before the upgrade. Neither have I tried changing the 'Check for new messages' setting from Automatically.
I'll post back in the next 24 and 48 hours to let others know if the issue has repeated or if I may have found another fix to the bugginess.