Continued corespotlightd process CPU overload issues

%CPU is a bit of an odd duck in today's world of multi-core processors. It made more sense when we were doing things on single-core architectures like the VAX (yes, I'm that old). Back then a process chewing large amounts of CPU was pretty obvious. In a multi-core architecture, a process can simply get allocated to a different core when it gets switched out, resulting in the appearance of using "more than 100%" of a CPU. (The core allocation logic is opaque to most human beings - and I'm not exactly sure what the kernel / hardware interaction looks like there - it's been a few moons since I did any amount of kernel work)

Top in its default form is a bit aggressive, resampling every second. I tend to use the command "top -o CPU -s 5" to make it a little less hungry.

Follow-up to my earlier observations:

Yesterday, after using my M1 MBP for several hours with no issues to do the same kind of document editing use-case that I had going before, I decided to turn off Apple Intelligence on my M3 iMac, and so far it although I do see corespotlightd occasionally spiking in CPU, it comes back down in a reasonable amount of time, and is not causing random freeze conditions, nor has it caused a CPU panic at this time.

I have a follow-up call with Apple Support tomorrow morning. (I started squawking about this with them last week when the problem resurfaced after a re-install of the OS).

Correlation? It seems concerningly probable that something to do with Apple Intelligence integration is related to this issue.

I was also convinced that this was related to having a large Pages file open. But then, after archiving last month's 35MB+ file and starting a new, empty one, I had the same issues that would not go away, even if I quit Pages.app and waited. It was making my Mini virtually unusable for writing (my main task) so I switched to my iPad, rebooted the Mini and started running Clean my Mac routines on it. No significant change. corespotlightd had shot to the top of the usage list within a minute or so of reboot and stayed there. By this time, I had two terminal widows open alongside Activity Monitor and I was watching what was going on carefully ('-s 5' to slow things down and '-o state' to list stuck processes at the top). At various times I could have up to 20 stuck processes. One of them was launchd which led me to some further research and the idea of disconnecting my Time Machine Drive. Within a few minutes corespotlightd had vanished from the CPU usage list. Reopening the previously opened Pages file was fine until I swapped to another programme and then corespotlightd returned. I saved and quit Pages and it went away again. I'm including all this info in case anyone else is having the same issues as I am. Maybe try disconnecting the time machine drive? (It seems a bit nuts, I know Spotlight isn't supposed to index that drive.)

Thanks for your posts! I have large Pages files on my M2 MB Air and Apple Intelligence turned on. Editing those docs is when I first noticed the problem that eventually led me to corespotlightd high CPU usage. I’m wondering if Writing Tools in Apple Intelligence is the culprit. I talked with Apple Support yesterday (NYE) and made it to the point in troubleshooting where we identified it as a User issue and not system wide. We progressed to a system re-install, which you already know doesn’t work. I’ll talk with Support again tomorrow, but in the meantime, I’ve turned off Apple Intelligence on my Mac to see if it will also calm down

Turning off Apple Intelligence on my M2 MBA macOS 15.2 stopped corespotlightd from hogging the CPU, dropping the process from in excess of 250% of CPU to 0.0 % within 8 hours. However, when I went back into my Pages doc (currently 1.2 MB, stored on iCloud), I continued to experience the spinning rainbow wheel, although less frequently and for less time. I've also experienced hesitation in my other apps (including while typing this post!), but no spinning wheel. I've talked with a member of the Senior Support Team at Apple and have arranged for them to collect data for Engineering to evaluate. That happens this coming Tuesday (my schedule doesn't fit with theirs until then). As much as we all don't like it, it appears it's a problem we'll have to live with for a bit until Engineering can figure out what's happening

I've had similar problems with all four of my Macs (a Mac Studio with an M1 Ultra, a MacBook Pro with an M2 Max, an iMac Pro, and a 17-inch 2020 iMac with an 8-core Core i7). On all four systems, I noted sometime around when 15.1.1 came out that corespotlightd would frequently top the list of processes, using anything from 100% CPU all the way to 750%(!) of CPU (on the iMac), causing the fans to spin up on the Intel systems to annoying levels. Updating to 15.2 did not resolve the issue for me, and in fact may have worsened it.

I did notice that corespotlightd calmed down quite a bit simply by closing a large Pages doc (200+ MB), and even more reliably by simply closing Pages on systems where I wasn't actively editing documents (at least, documents that are synced to iCloud). Corespotlightd will still occasionally ramp up to 100+% of CPU, but it won't stay there indefinitely, and most of the time it will be under 50%.

But another more serious issue: I found that after I installed 15.1.1, I could no longer back up the Intel systems via Time Machine. If I tried, one or both situations would arise: (i) a Time Machine backup would be "preparing" for hours or even days; and (ii) attempting such backups would frequently lead to repeated kernel-panics. The only resolution for this latter issue I have found is to disable Time Machine backups entirely. Even starting a backup on a freshly-erased backup drive with no existing Time Machine backups would still lead to the same behavior vis à vis interminable "preparation" of backups and repeated kernel panics.

One thing occurred to me during all of this trouble-shooting: I believe that Time Machine relies on Spotlight to identify files which have been modified since the last backup. I think it may be there is some bug in corespotlightd that aside from consuming vast system resources, also leads to complete (albeit short-lived) system freezes where they don't cause a kernel panic, and also interferes with the proper operation of Time Machine.

Just for, I don't know, comic relief maybe: I came home today to find that corespotlighd was using 1,400% of available CPU time on my 8-core iMac Pro.

How the system avoided crashing completely is a bit of a mystery.

Mitch Stone wrote:

Interesting. Checking these folders on my M2, I find that the size of the CoreSpotlight folder is 37GB, but no file within it is even a megabyte in size. The SpotlightKnowlegeEvents folder clocks in at 463 MB. It contains far more subfolders, so it is difficult to figure out where this data is hiding. Has your performance been improved by deleting the contents of these folders?

And another FWIW: Since the last time I deleted the Spotlight plist about a week ago, I have had no corespotlightd process issues. So I do think this is worth trying.

As noted, this issue does not appear to be as severe on Apple Silicon systems as it is on Intel systems. My M2 Max MBP has about what yours has in the CoreSpotlight folder: about 42 GB. But many of the files in there are tens of megabytes, some over 100 MB (but I edit large Pages documents, and others have pointed out that this can exacerbate the problem).

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

I haven't tried deleting the spotlight .plists yet (there are appear to be five plist files with "spotlight" in the filename, at least on Intel systems), but I do see that Spotlight is still writing large amounts of metadata to the user library folder, 8.4 GB in just the last two hours. If that trend continues, I'l try deleting the .plist files as well.

ericmurphysf wrote:

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

Progress report: since deleting all this metadata, not only has the system seemingly stopped writing more of it (or at least slowed way, way down in adding to it), but corespotlightd has been at barely above 0% in terms of CPU utilization. I'm not sure this is a magic bullet in resolving this issue, but so far it seems to have resolved a plethora of issues I've had with my Intel Macs.

CaptainJoy wrote:

I'm wondering why I was affected and others not? One unusual thing about me is that I was upgrading from a 2014 Mac Mini. Maybe the jump from Monterey (OS 12) to Sequoia (OS 15) when migrating my old stuff over via Time Machine had something to do with it?

How heavy a user of Pages are you? What I can say for a virtual certainty is that editing large (>10 MB) Pages documents massively increases the amount of metadata saved to the folders inside ~/library/metadata/ that are concerned with Spotlight indexing. If I don't have any Pages documents open on a Mac, metadata in these folders might grow by a couple of hundred megabytes over a span of 24 hours (which is still a lot, but it's not insane). With a large Pages file open, I might see an additional gigabyte of new metadata over a period of less than an hour.

My guess as to what's happening here is that, when you edit a Pages document, it's saved automatically to storage (and if iCloud sync is turned on, it's also saved to the cloud). But instead of the various Spotlight indexing processes just indexing the new content in the file, they reindex the entire file, and save the additional data alongside of existing metadata rather than replacing it. Consequently, a single 10 MB file might result in tens of gigabytes of metadata being saved if you spend a lot of time editing that one Pages file.

As evidence in support of this hypothesis, I've opened up several of the larger (tens to hundreds of megabytes) of the .journal files that are saved in these metadata folders in TextEdit. A simple search shows that the larger ones contain tens of thousands of references to the specific Pages file I happen to be editing.

That seems pretty close to a QED in terms of what's happening.