Two things I discovered while investigating this same issue:
32-bit Spotlight Importers
These seem to be one of the reasons mds and mdworker (the system processes behind Spotlight) were tripping up and failing to complete the index. (And might explain the mysterious "Unable to talk to lsboxd" messages.) Since mds and mdworker are 64-bit processes, it probably couldn't work with the older 32-bit Spotlight Importers (I found three on my system).
To find out if you have any, use Terminal.app and enter this command: mdimport -L
That'll give you a list of paths where you can find each of the Spotlight Importers (some are embedded in third-party applications). Use the Finder "Go to Folder..." command to open the enclosing folder of each Spotlight Importer. In the Finder, "Get Info" on each to see if it says "(32-bit)". I moved these to the trash and tried the process suggested above and it seemed to work better.
Rebuild Mail.app mailboxes
This was what finally got Spotlight to index email that hadn't been included in previous attempts, even following the processes suggested above and elsewhere. Only after Mail.app rebuild the mailboxes did Spotlight begin to return search results from those locations.