Leopard's Spotlight doesn't search Mail messages

This solved the search issue for me. How many of us with mail search issues did an upgrade install to Leopard?

I chose Archive and Install for my original install of Leopard and I still have the Mail search bug.

Hi Mick. Glad it worked for you. Unfortunately, for me, this actually made things a bit worse.

Not only did I do the things below, but later also retried the same thing plus deleting Spotlight indexes on the hard drive and in the user library folder. I let Mail and Spotlight take all night, and the problem remains.

The new annoying problem this caused is that when I re-imported the Mail folder messages from the desktop, it skipped a few hundred of them, in a few dozen separate mail folders. i.e. if I had 2000 email messages in 20 Mail folders, after import I have 1900. Stupid Mail did not even at least give me an error list and say the following specific .emlx messages were not imported.

Manually investigating and comparing a particular folder, e.g. 190 messages in imported 'Bahamas Trips' mailbox and the 200 messages in the 'Bahamas Mail' Messages folder in the Mail folder sitting on the desktop, I could pore through the list and find the 10 that were not imported by Mail using the method you described.

What is doubly strange is that each of those 10 'un-imported' messages can be clicked on and opens up a Mail message window just fine, and readable. That means the .emlx is not corrupted or something else, but somehow using this method Mail skipped a few hundred. It's going to be a pain to go through each message folder and find what messages are missing and find some way to get them back into Mail. (Dragging them to Mail, re-importing them again, all are not working).

Just wanted to share my experience if others have the same problem.

Imran

Terrell: Mike posted a solution that involved rebuilding the Launch Services database. The Terminal commands required to do so are not on the list of things you've tried. Could you look at all MikeW's posts in this thread and locate his messages of 16th November 2007? They've worked for me. As a test, go to your mail folders in the Finder and see if the items ending with "imapmbox" or "mbox" appear as folders that you can browse without needing to show package contents. The problem in my case was caused by older versions of Mail (in my case, on external drives) present when Leopard built its initial indices.

Hi Bahi, and welcome to the Apple Discussions.

I read Mike's post of 16 Nov. I do not use imap in Mail, so have no imapmbox files on my machine. The folders which contain mbox are available as browsable folders, and I can see the contents.

After completely erasing my HD and restoring from my Time Machine backup, Mail searches started working again. Whatever was broken, was fixed by this.

Terrell Smith wrote:

After completely erasing my HD and restoring from my Time Machine
backup, Mail searches started working again. Whatever was broken,
was fixed by this.

Excellent. I think I'm getting a good idea of what's going on to cause this. I'll submit a bug report but I've been in touch with someone who already did so and was contacted by Apple with further requests for information - they're clearly on it. In my case, the trigger was an unfortunate upgrade bug to do with the way launch services associates data files with applications; the result is badly assigned data types (UTIs) that confuse Spotlight's plugins - in this case, the Mail plugin called Mail.mdimport. Anyway, all's well that ends well. Glad that the re-install and import fixed it.

Hi Bahi,

When you say "Apple is clearly on it" that might be one incidence. We all need to send them feedback on this, so they really get on it. In another posting, I describe how two Apple geniuses at an Apple store spent a very long time on it, and were unable to fix it. And as evidenced by many posts here in these discussions, there are a number of people with this problem.

Apple does read Feedback (though they rarely reply), and with all of us who are experiencing this sending them feedback, they will eventually get a fix.

So, if you have this issue, please use the link in Mail and send Apple feedback. Using the link in Mail also sends Apple information to help them (hopefully) better isolate the problem. Thanks!

Terrell Smith wrote:
When you say "Apple is clearly on it" that might be one incidence.
We all need to send them feedback on this, so they really get on it.

The most effective way to help at this stage, now that engineers are following up on the existing bug reports, is by submitting more bug reports to make sure that all the various instances of this thing are caught. Early on, feedback (through forums but especially via Apple reps) detailing the pain that a bug causes is really important, because software engineers' priorities are decided in part by the amount of benefit a bug-fix would bring. For example, the Active Directory binding issues in Leopard were very visible and high profile and were clearly getting reported loud and very clear (and in no uncertain terms) in the corporate and education markets, as evidenced by the feedback reps were getting. 10.5.2 contained a fix.

In another posting, I describe how two Apple geniuses at an Apple store
spent a very long time on it, and were unable to fix it. And as evidenced
by many posts here in these discussions, there are a number of people
with this problem.

Yes. It's obscure, it involves at least one bug and requires a relatively unusual set of circumstances before it's seen. It affects a very small proportion of the people who upgrade. These bugs are the hardest to reproduce. If you're a software engineer who can't reproduce the problem, you're a software engineer who can't fix the problem. Reading more reports by people who say "I have a problem" might indirectly help get an engineer assigned to a problem but it doesn't help the engineer fix it. We're at stage two now.

Apple does read Feedback (though they rarely reply), and with all of us who are
experiencing this sending them feedback, they will eventually get a fix.

At this point, with software engineers already contacting people who've submitted detailed bug reports about this exact issue, we can be sure that Apple knows of the problem in at least some of its incarnations. The entire fix in this instance cannot come from Apple geniuses at stores - it needs to come, at least in part, from the engineering department - any other fixes (and it is possible to fix all the symptoms) will be immediately broken by a reoccurrence of the circumstances that triggered the original issue because there's an issue in the shipping code that needs to be addressed.

So, if you have this issue, please use the link in Mail and send
Apple feedback. Using the link in Mail also sends Apple information
to help them (hopefully) better isolate the problem. Thanks!

Good, detailed bug reports are the best way to help software engineers. Feedback to Apple reps with a clear indication of cost to you or your institution (and, of course, the effect it's having on the orders you're likely to send Apple's way) is the best way to get a bug fix's priority raised. Both steps are required. Those steps are echoed and repeated regularly by Apple engineers in almost all the technical Apple mailing lists. Posts here may be helpful in an indirect way but I don't imagine that the emotion on display in some of these forums helps people evaluate priorities. Everyone thinks his or her problem is critical and that Apple must get onto it straight away - not helpful.

Anyone who can reproduce this problem on more than one machine must please submit a [bug report|http://developer.apple.com/bugreporter> to help an engineer reproduce that particular variation of the issue. Had they seen this problem is development, engineers would have caught this before Leopard shipped. A problem that a software engineer cannot easily reproduce is one (s)he cannot fix and the circumstances that trigger this one are quite obscure.

Not everyone will be able to write a clear, dispassionate [bug report|http://developer.apple.com/bugreporter> but those who can should do so. Those reports are accessible by the software engineers who can fix the problem. Free ADC membership is required and it can be attached to the e-mail address you use to post to this forum.

Message was edited by: Bahi

I have found some interesting things which may be helpful in narrowing down the problem.

1. In addition to Mail, I find Spotlight's Command-space search does not show Document results if I search for particular words, so it's not just Mail messages my Spotlight problem is missing, it is also missing words inside non-Mail documents.

2. I am using VileFault, I mean FileVault. How many others of us are using that, in case it has a bearing on the problem?

3. This was interesting. I logged in as Root user, and mounted my FileVault sparseimage. Guess what, Spotlight under root user quite easily found all my Mail messages, documents, etc. for any search term I put in the command-space blue search box. That means the Spotlight index on the hard disk and the sparseimage is fine, but somehow the user "imran" is unable to get to that data, while root can.

4. This was another interesting thing. I used Onyx to make invisible files visible on my PowerBook G4 and on the MacBook Pro on which Spotlight works fine for non-root users. I was surprised to see that the problematic machine shows permissions for Spotlight giving "system" read/write access, but on the non-problematic MacBook Pro, the equivalent Spotlight file shows "system" having 'Custom' permissions. (I was unable to see what is Custom, because in Terminal I could not get results from a ls -l .Spot* command.)

So, as far as I can tell, something causes the permissions on the spotlight indexes to get changed, which Onyx and Apple Disk Utility are unable to fix. The index/indices themselves work fine for root, etc, but not under the main user. If someone can comment on how to change the permissions on Spotlight index 'files' to the correct 'Custom' permissions, we may have some luck.

Doing a full TimeMachine backup and restore are not options that FileVault users can exercise without out even MORE aggravation and time wastage.

Looking forward to your suggestions.

Imran

Everyone - Please note that reporting the problem here on these discussions is NOT the same as submitting feedback to Apple. Apple rarely reads these discussions. These discussions are for users to help users.

You need to submit feedback to Apple. The way you do that is either through this link (you don't need to be a developer): http://www.apple.com/feedback/
or (better) through the "Provide Mail Feedback" link under the Mail menu.

Imran,

I've never used FileVault, so I doubt that it's related.

Since activating Filevault I have been unable to search the entire message, (non admin account).

The system was working fine before the fielvault switch was flipped on.

Richard.

Hi all. I have finally gotten my Spotlight working. It was not easy. These are steps from memory about a week ago, so use caution, and your own research and common sense. I am not liable for what your experience is.

1. I have always felt it was a permissions issue somewhere (since root user can find the Mail messages in Spotlight but user imran could not).

2. FileVault is VileFault but not the culprit in this case. (May be a contributor but others have the same problem without using FileVault).

3. Having FileVault enabled did make my problem solving more complex and time consuming. I had chosen to disable FileVault. But, guess what, that is a bigger pain than you can imagine, but will be discussed elsewhere.

4. You may not want to be as paranoid as I was, but this is what I did. Basically, I used SuperDuper and Carbon Copy Cloner on two separate external 160GB drives to make clones of my laptop drive. Yes, I am paranoid. I tested both to work as boot up drives. THEN, I cloned the 160GB drive to a 500GB external. Why? Because, unfortunately, FileVault needs a lot of disk space to decrypt the files, and throw in the fact that the sparseimage also assigns itself a max size at the time of creation, which made it impossible to decrypt on the laptop itself or the 160GB external clones. I had to let it finish decrypting on the 500GB external unit as the startup disk.

5. I then let Spotlight run overnight off of that external startup drive. (That way I figured I could play with different tests and if all failed, at least the laptop internal drive would be no worse than before and I could continue to use it with broken Spotlight).

6. I also threw away the Mail mailbox envelope index type files, so Mail reimported 20,000 emails along the way to give Spotlight the best chance to work.

7. I ran Mail the next day and... DARN... no, Mail still did not show Message Contents/Spotlight. Which shows that FileVault was not the culprit.

8. NOW, I went for the real test. I booted off one of the other external drives. I created a user "anwar" with admin privileges. I removed my main "imran" account's admin privileges and then deleted the "imran" account. Then I recreated the account "imran". I then logged in as imran. I then manually dragged over all the "imran" home directory decrypted files from the home folders on the 500GB drive. (I had to drag the contents of folders to equivalent folders. i.e. I selected-all content in 500GB/User/Library and dragged that to the 160GB/User/Library as simply overwriting folders by folders would not work because of special permissions that are associates with them. (Are you still with me!?)

9. Finally I had copied over every sub-folder and file from the user home decrypted drive to the newly created user home folder. I then ran Mail after removing the Envelope Index etc. again (just to make sure I had a clean slate). 20,000 emails were imported.

10. I then let Spotlight run overnight... next morning, I ran Mail, and I could actually search for email messages by Content of messages!!

11.... EXCEPT... now, there was no text showing in the preview pane in Mail nor would opening the message show any text. DAMMMMMIT! Now I can search for messages but can't read them?

12. So, I had to manually select EACH Mail folder AND sub-folder On My Mac (which represents some .mbox file in User/Library/Mail/etc.) and click on Rebuild in Mailboxes.

13. And, finally, not only are all Mail messages being searched but also displayed correctly.

14. Now that I had this tested and running, THEN I finally used SuperDuper or Carbon Copy Cloner (I forget which one) and cloned the external 160GB that I had been doing all this work on right on top of the internal laptop 160GB drive and it has been working fine for a week.

Sorry for the long post, but this will ensure you see what steps may have solved the problem and what shortcuts you may want to try. I still think deleting the user account and recreating it set up some permissions correctly. I would have preferred not to have to manually drag user folder sub-folders to the new account (because among other things I had to reenter the license info on many programs when I ran them) but at least it works.

Hope this helps. Use caution, I had 3 external sets of drives to play with to ensure I did not accidentally lose the internal and corrupt the external and be completely shafted. Use your own discretion. If you think installing the OS again and restoring is easier, feel free to try that. 🙂

Imran

1. I have always felt it was a permissions issue somewhere (since root user can find the Mail messages in Spotlight but user imran could not).

That was not the case for me, I was running as root and Spotlight saw no Mail messages, nor could Mail search them on content.

a- Clearly different people have had different experiences. I stated mine.

b- How did you check if Mail was able to search messages' content when you were logged in as root? Did you import your username Mail folders into Mail running as root and wait for Spotlight to index or some other way? Do use Mail and get emails on the root account that you also see Spotlight fails to work for?

c- What worked for me may not work for others. So, how did you solve the problem? I am sure a lot of people would love to hear that.

Thanks.

Imran