Thanks Douglas. I was able to successfully troubleshoot the mds process.
After 3 days of trying to find the offending file(s) I found it. Basically, what is happening is that the Spotlight process (mds) and it's workers (mdworker & mdworker32) are getting choked with certain files (not sure why, but they should've designed this differently).
From scouring various sites I was able to troubleshoot Spotlight and rebuild the index successfully. Here's what I did:
1. Downloaded Spotless. This will help you "see" the index file being built.
2. Enable "root" user to troubleshoot in Terminal. You don't really need to do this, however, to find directories and files that belong to root, you need to be signed in as root.
3. You need to stop the mds process. You can stop mds using Spotless or run this command in Terminal (you'll need to have administrator or root access).
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
4. Once mds is stopped, make sure you're at the / level or the directory that you are trying to rebuild. Delete the following directory with this command.
sudo rm -rf /.Spotlight*
5. Turn back on Spotlight and start to re-index.
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
You can restart using Spotless or by commands with Terminal. Leave Spotless open to refresh and see the index grow.
6. As soon as you've started Spotlight, it will start to build the index. Now open a separate Terminal window and run the following command:
sudo fs_usage -f filesys mds mdworker mdworker32 | egrep "open"
(this command will display the files that are being added to the Spotlight index and it will scroll)
Now you'll have to wait to see where Spotlight is choking. For my machine, after about 45 minutes, it started hanging on the following directory:
Support/MobileSync/Backup/8fasdwecdsf….
How do you know it hangs? Well, mds was maxed out at 99%, the Terminal window showing progress of building the index didn't show more files being added (in other words it stopped scrolling), it seemed like a lot of blank space and Spotless didn't show that the index was increasing in size (it was hanging around 490MB).
Once I knew for sure it was hanging, I scrolled through Terminal until the last legible entry and that's where things looked strange. There was a lot of long character strings with no direct correlation of a file name or folder. The last entry before this junk was related to the above directory Support/MobileSync/Backup/8fasdwecdsf…./ that's where I decided to concentrate my findings.
I stopped the mds process using the above methods in Terminal and I did a search on the web and found this forum (http://www.philmug.ph/forum/showthread.php?t=66067) explaining that this directory was for my iPhone back up. When I viewed the files (as root), I could see there were a lot of files. Since my back up on the phone was recent (that's when I started having problems with Spotlight as I looked back), I decided to delete these files through iTunes > Preferences > Devices > Device Backups. I wasn't sure if deleting these backups would help Spotlight, but I knew that it was hanging up Spotlight based on my Terminal log file.
After deleting these files, I restarted Spotlight (Terminal or Spotless), turned on the sudo fs_usage command (Step D) and watched the index get rebuilt again. This time as it approached 490MB I was aware that it could choke, however, it passed and continued to process more files and the index stopped around 620MB, the mds process stopped normally and Spotlight up and running again!!!
Here are some other sites that were helpful with my unix commands (I'm a little rusty):
http://www.macgurulounge.com/spotlight-stops-indexing-files/
http://forums.macnn.com/90/mac-os-x/380957/spotlight-permenantly-estimating-inde xing-time/
https://discussions.apple.com/thread/2297570?answerId=10954614022#10954614022&messageID=10954614
Good Luck!!!
