Repeated mds/mdworker segmentation faults: use DTrace to diagnose?
Hi --
Ever since I installed Leopard (via "archive and install"), Spotlight won't stop re-indexing. I'd say it totally reindexes itself 2-3 times a day. That's been pegging my processor and really killing my battery life.
In the console log, I see lines like this:
12/14/07 1:41:33 PM com.apple.launchd[1] (0x10cc20.mdworker[739]) Exited: Terminated
12/14/07 1:41:33 PM com.apple.launchd[1] (0x10ca00.mdworker[738]) Exited: Terminated
12/14/07 1:42:17 PM com.apple.launchd[1] (com.apple.metadata.mds[736]) Exited abnormally: Segmentation fault
I'm a curious fellow, and so I started nosing around seeing if I could figure out what files were causing the crash -- perhaps Spotlight is choking on something. So, I typed the following DTrace command in Terminal (while logged in as an admin user):
sudo dtrace -n 'syscall::open*:entry /execname == "mdworker" | execname == "mds"/ { printf("%Y %u %s %s",walltimestamp,pid,execname,copyinstr(arg0)); }'
...which produces a nifty, scrolling terminal window that contains entries like this:
1 17600 open:entry 2007 Dec 14 13:14:23 678 mdworker /Library/Logs/CrashReporter
What I've been trying to do is see if I can correlate information that DTrace is providing with information in the console log. But the results seem to be all over the map. I must be missing something, because there's always a 15-second gap between my last DTrace entry and the console "segmentation fault" message. Example:
Here's an entry from the console log:
12/14/07 1:20:23 PM com.apple.launchd[1] (0x10cc20.mdworker[679]) Exited: Terminated
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker
And here are the last entries from DTrace shows for pid 679 (mdworker):
/Users/arj/Documents/Andy's Personal Notes.nb/index/CapitalizedWords.nbindex
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Documents/Andy's Personal Notes.nb/index/Numbers.nbindex
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Documents/Andy's Personal Notes.nb/index/Keywords.nbindex
0 18386 open_nocancel:entry 2007 Dec 14 13:20:00 679 mdworker .
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /dev/null
1 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Library/Application Support/AddressBook/Metadata/.info
1 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Library/Application Support/AddressBook/Metadata/.info
I've done this sort of analysis about a half-dozen times. There's no rhyme nor reason to the files immediately preceding the seg fault, and there's always an ~15 second gap.
I don't really know DTrace that well, so I could use some pointers. Anybody got any good ideas? Or are there other ways to figure out what's giving Spotlight fits?
Ever since I installed Leopard (via "archive and install"), Spotlight won't stop re-indexing. I'd say it totally reindexes itself 2-3 times a day. That's been pegging my processor and really killing my battery life.
In the console log, I see lines like this:
12/14/07 1:41:33 PM com.apple.launchd[1] (0x10cc20.mdworker[739]) Exited: Terminated
12/14/07 1:41:33 PM com.apple.launchd[1] (0x10ca00.mdworker[738]) Exited: Terminated
12/14/07 1:42:17 PM com.apple.launchd[1] (com.apple.metadata.mds[736]) Exited abnormally: Segmentation fault
I'm a curious fellow, and so I started nosing around seeing if I could figure out what files were causing the crash -- perhaps Spotlight is choking on something. So, I typed the following DTrace command in Terminal (while logged in as an admin user):
sudo dtrace -n 'syscall::open*:entry /execname == "mdworker" | execname == "mds"/ { printf("%Y %u %s %s",walltimestamp,pid,execname,copyinstr(arg0)); }'
...which produces a nifty, scrolling terminal window that contains entries like this:
1 17600 open:entry 2007 Dec 14 13:14:23 678 mdworker /Library/Logs/CrashReporter
What I've been trying to do is see if I can correlate information that DTrace is providing with information in the console log. But the results seem to be all over the map. I must be missing something, because there's always a 15-second gap between my last DTrace entry and the console "segmentation fault" message. Example:
Here's an entry from the console log:
12/14/07 1:20:23 PM com.apple.launchd[1] (0x10cc20.mdworker[679]) Exited: Terminated
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker
And here are the last entries from DTrace shows for pid 679 (mdworker):
/Users/arj/Documents/Andy's Personal Notes.nb/index/CapitalizedWords.nbindex
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Documents/Andy's Personal Notes.nb/index/Numbers.nbindex
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Documents/Andy's Personal Notes.nb/index/Keywords.nbindex
0 18386 open_nocancel:entry 2007 Dec 14 13:20:00 679 mdworker .
0 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /dev/null
1 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Library/Application Support/AddressBook/Metadata/.info
1 17600 open:entry 2007 Dec 14 13:20:00 679 mdworker /Users/arj/Library/Application Support/AddressBook/Metadata/.info
I've done this sort of analysis about a half-dozen times. There's no rhyme nor reason to the files immediately preceding the seg fault, and there's always an ~15 second gap.
I don't really know DTrace that well, so I could use some pointers. Anybody got any good ideas? Or are there other ways to figure out what's giving Spotlight fits?
MacBook Pro, Mac OS X (10.5.1)