Smart mailboxes are empty in Mavericks mail

I think I am getting an understanding about Smart Mailboxes and spotlight in 10.9. In 10.8 they were quite independent, in fact I had spotlight disabled for mail, as there are better ways to search. But in 10.9 Mail is extreamly dependent on Spotlight. All of the flagging, tagging, searching done in mail is in fact done by spotlight. That includes filling the contents of the Smart Mailboxes as well. It appears that Smart Mailboxes are no more than cashed search presets.

So here I am in 10.8 with mail and most of Spotlight disabled, and along comes 10.9. It appears that the 10.9 installer lays down a new version of Spotlight, creates default preferences in two locations, but uses the preference from the 10.8 version. So when I go in to System Preferences for Spotlight, it says everything is enabled. But if I search for an email in Spotlight, it finds no searchable email. All of the Smart Mailboxes are empty as well.

I look in activity monitor and find no activity for spotlight, so I am pretty sure its not updating it's database, in fact I could not find the spotlight database as an admin. I find some importers, the preferences and such but not the database it's self. So I do a su, and as root, I find two copies, a V1 and a V2. Both about the same size. Which is interesting as if Spotlight was using the new preferences, it should have creaated a database several times larger.

After looking around the system and finding nothing else of note, I decided to take the bold move deleting all Spotlight preferences and databases, and rebooted. After it came up, Spotlight mentioned it was creating a new database, and the spotlight system preferences was at the default setting (all enabled). So I added a few folders for privacy (and testing), and let it do it's thing. After a few hours, some of my smart Mailboxes started to fill up, and after Spotlight completed it's work, everything is working correctly.

Smart Mailboxes are completly dependent on Spotlight for it's functionality. I wish Apple would mention that in Spotlight, and maybe remove the checkbox (or give us a warning) to disable Spotlight for mail. If you are having problems with Smart Mailboxes in 10.9, I would look in Spotlight first. There may be other issues, but from my experiance (and those mentioned in other threads) this cannot be overlooked.

jsoden wrote:

I think I am getting an understanding about Smart Mailboxes and spotlight in 10.9. In 10.8 they were quite independent, in fact I had spotlight disabled for mail, as there are better ways to search.

Smart Mailboxes have always be dependent on Spotlight. I'm not sure how you had it set up, but it has always used Spotlight.

Well, I had the System drive (Macintosh HD) in the Spotlight Prevent search in these locations and all the Search results not selected (except Apps). I had no problems with Smart Mailboxes until 10.9

It's quite possible that it used both. The messages are stored in a database which could be quickly queried for basic stuff like subject, sender, etc. The content would have to be indexed by Spotlight. So, it may be true that it has changed to using only Spotlight, but it may be that your index (database) is corrupt.

You could try rebuilding the mailboxes.

Another way to do that is to "vacuum" your Envelope Index: http://brettterpstra.com/2012/09/15/vacuuming-your-mail-app-index-still-works-wo nders/

Finally, you can delete the Envelope Index and it will be recreated.

For any of those, I would suggest making sure you have your email backed up.

One of the interesting things was I could not get Spotlight to re-index. I changed the Spotlight preferences to add and remove the system drive, I deleted the preferences, both in the user and root libraries. It took a deletion of the full database to trigger a re-indexing. I think that there would be an easier way to create a new index, but in my case the standard way did not work.

I agree that the likelihood of my index being corrupt, is high.

I did try to create a new Smart mailbox, rebuilding existing mailboxes, etc. None of which were effective.

That link is interesting, I will have to experiment.

I confirn that I had spotlight disabled. Once I anabled it and it finished indexing smart mail boxes become work properly.

I think that it is strange that I cann't disable spotlight.

I re-indexed spotlight by adding and removing Mac HD in the privacy list of spotlight and smart mailboxes seem to work better.

I added and then removed the Mac HD in the privacy list of spotlight. Spotlight took 4 hours to reindex.

After that process, most of my smart mailboxes are still NOT working.

This is really frustrating since Smart Mailboxes were a critical part of my "system" for managing the very large amount of email I receive on a daily basis. Without Smart Mailboxes it is very difficult to make sure that a priority email doesn't slip through the cracks.

hopeisnotastrategy wrote:

I added and then removed the Mac HD in the privacy list of spotlight. Spotlight took 4 hours to reindex.

How large is your hard drive. Four hours seems excessive. Might indicate something is corrupted on the drive.

Robert, I experience the very same problem, with smart mailboxes that refer to other smart mailboxes. With the very same messages in console. Unfortunately recent the update of Mail.app hasn't helped.

Hi everybody, it seems after reading this thread, I was able to fix my Smart Folders in OS X Mavericks. I waited for Apple to release the Mail.app update, which they just did, but that did not fix my smart folders so I have to take drastic action. I was getting tired of it taking almost a minute to load my Smart Folders. I was also having trouble with search in Mail.

That said, I was specifically having problems with 4 of my Smart Folders: Today, Yesterday, Past Week, and Unread. All canned filters in Mail. The rest were custom, and loaded fairly quickly, but I was still having search issues.

What peaked my interest in this thread was about how Smart Folders relied on Spotlight to be effective and some people were trying to use the Spotlight System Preference to control this. I did not have Mail & Messages unchecked in my Spotlight prefs.

So, I recalled that I had used an app for OS X before called, Onyx, which allowed me to delete Spotlight index, though I had never used that option before. I've used Onyx for other things many times previously on older versions of the big cat, and indeed, it is available for Mavericks, so I downloaded it again, and used it this time to remove my Spollight index (just search around in the features). After you click Execute, it runs a script, then restarts your Mac . After starting up, your Mac will reindex the Macintosh HD, recreating the Spotlight index. It took a few hours on my 17" MBP (last gen).

This method worked VERY WELL for my Smart Folders: Today, Yesterday, Past Week, and Unread. Mail.app is really fast again, has useful search, and actually feels like a real Mavericks update.

Give it a try! Let me know if it works for you, too.

Message was edited by: Adam Blainey

I just ran into this problem. In my case I keep my email in an encrypted DMG virtual disk, which I open once I've booted up. Mail works with this arrangement just fine and it keeps my mail encrypted without me having to encrypt the entire drive.

With the upgrade to mavericks the smart folders stopped working. Digging around this thread (and others) I suspected that the problem was due to a lack of spotlight indexing. This hypothesis was supported by the fact that no emails turned up in my spotlight search results.

(It's very likely this was the case pre-mavericks, as I never used that feature. Furthermore I was always annoyed that searching email contents never seemed to show any results, although searching on headers such as subject or to did work.)

I tried the advice to add then remove my dmg folder from the privacy pane in System Preferences -> Spotlight, but this didn't seem to work. Not just did no emails show up in my email, but there was no spotlightish activity in my activity monitor. Another sign that nothing happened was by looking at the spotlight data. Spotlight stores its data in /Volume/dmg-name/.Spotlight-V100. Inside this volume there are Store-V1 and Store-V2 directories. On my main hard drive the Store-V1 was empty but Store-V2 had a folder with a name that looked like a hash. No such folder was present on my dmg.

I then tried using the mdutil command in the terminal. I did this

mdutil -d /Volume/dmg-name

mdutil -i on /Volume/dmg-name

At that point everything kicked into life, Activity monitor buzzed with mdworker tasks, a hash named file appeared in .Spotlight-V100/Store-V2, and - most importantly - my smart folders started filling up with email.

I hope this overly long explanation of my actions helps someone else with their missing messages.

Exactly this worked for me! I migrated via Time Machine from my MacBook Air to MacBook Pro 13" Retina and everything worked out fine except that in Mail my Search didn't work anymore and my smart mailboxes were empty. After adding and removing Mac HD in the privacy list of spotlight (System Settings) everything worked fine again! 😁

Yay, I found the bug that bugged me!

Do your smart mailboxes have very many rules? Or do you use a rule like "Sender is member of group" with a group that contains very many people?

I saw that smart mailboxes only work up until all rules for one smart mailbox combined count up no higher than 1024 (yes, the bug is that lame). Note, some rules like "From contains" actually generate 2 Spotlight conditions ("email contains" and "name contains"). And if your smart mailbox has rules based on other smart mailboxes their rules increase the count, too.

You can check this yourself: Open Console.app, look for a message starting with "Unable to create Spotlight query for expression". Copy that message using CMD+C to an application that lets you search within text. Then search for all occurrences of the symbols != and == in it. They surely amount to greater than 1024.

In my case I have 2 groups in Contacs.app that each have about 400 people in them. When using these groups as smart mailbox rules they worked up until the moment when I combined them with many other rules.

I already reported the issue to Apple. Hope they can increase the Spotlight search item limit of 1024.

I wonder if mine problem is related in any way!

Since I installed OS X Mavericks I am losing content of email messages. I can open received email messages from the basic INBOX but - ONCE I HAVE MOVED THEM FROM MY INBOX TO A SEPERATE MAILBOX for filing purposes - all I am left with is the title eg,

As shown in INBOX:

open it and could see whole content

..then after move to File Mailbox:

open it and everything, apart from title and date = gone and irrecoverable.

Even moving it back to INBOX = content lost.

I've tried all the restoration from Time Machine etc with no joy.

If this is not related perhaps someone could re-direct me.