Continued corespotlightd process CPU overload issues

CaptainJoy wrote:

I'm wondering why I was affected and others not? One unusual thing about me is that I was upgrading from a 2014 Mac Mini. Maybe the jump from Monterey (OS 12) to Sequoia (OS 15) when migrating my old stuff over via Time Machine had something to do with it?

How heavy a user of Pages are you? What I can say for a virtual certainty is that editing large (>10 MB) Pages documents massively increases the amount of metadata saved to the folders inside ~/library/metadata/ that are concerned with Spotlight indexing. If I don't have any Pages documents open on a Mac, metadata in these folders might grow by a couple of hundred megabytes over a span of 24 hours (which is still a lot, but it's not insane). With a large Pages file open, I might see an additional gigabyte of new metadata over a period of less than an hour.

My guess as to what's happening here is that, when you edit a Pages document, it's saved automatically to storage (and if iCloud sync is turned on, it's also saved to the cloud). But instead of the various Spotlight indexing processes just indexing the new content in the file, they reindex the entire file, and save the additional data alongside of existing metadata rather than replacing it. Consequently, a single 10 MB file might result in tens of gigabytes of metadata being saved if you spend a lot of time editing that one Pages file.

As evidence in support of this hypothesis, I've opened up several of the larger (tens to hundreds of megabytes) of the .journal files that are saved in these metadata folders in TextEdit. A simple search shows that the larger ones contain tens of thousands of references to the specific Pages file I happen to be editing.

That seems pretty close to a QED in terms of what's happening.

CaptainJoy wrote:

Now that my Time Machine backups are working again, I will probably try the above the next time the corespotlightd process gets too big for its breeches. After less than a day, it's up to 21.41 GB. I wish I noted how big it was before I cleaned it out. Going forward, I request people let us know how big their CoreSpotlight folders are before they delete them.

I've been keeping notes on the four Macs I own, all of which have this issue to a greater or lesser extent. The two Intel systems (a 27-inch iMac and an iMac Pro) each had about half a terabyte of Spotlight metadata in the two folders in ~/library/metadata/ that are associated with Spotlight. I deleted the contents of those folders as noted above, and by the next day those folders had grown to about 11 GB on both of them. Since then, depending on how much I've been using Pages on those systems, they have grown by between four and twenty-five gigabytes a day. The iMac Pro is already up to 103 GB (after eight days since I deleted the contents of those two folders).

On the two Apple Silicon systems, an M1 Ultra Mac Studio and an M2 Max MBP, metadata has grown much more slowly, largely I think because I tend not to use Pages as much on those two systems. In the four months since I updated to Sequoia, those systems have accumulated about 22 and 37 GB respectively, and so I have never deleted the contents of their metadata folders (largely because metadata at that size does not seem to impose much of a load on the system).

Since the metadata folders have exceeded 100 GB on my iMac Pro (in barely over a week), I'm going to delete the contents of those folders again. As I was typing this, I saw corespotlightd CPU usage spike dramatically, and at one point the entire system halted for about fifteen seconds.

That can't be good.

AshkaTheMoltenFury wrote:

So in short: give the cleanup of the ~/Library/Caches folder a try. It might help and solve this high CPU usage of corespotlightd. Hope this helps anyone.

tl;dr — cleanup of the ~/Library/Caches folder did not work; trashing the contents of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents did work.

Folder/File Sizes Before "Fix"

• /System/Volumes/Data/.Spotlight-V100 at zero bytes
• ~/Library/Metadata/CoreSpotlight at 66.66 GB
• ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents at 9.59 GB
• ~/Library/Caches at 1.9 GB

I do not believe “Optimize Storage” is turned on

Disk Writing:

• kernel_task had written 7.32 TB
• mds_stores had written 954.75 GB
• launchd had written 535.32 GB (https://www.technewstoday.com/mds-stores-on-mac-high-cpu-usage/ recommends disabling Spotlight—which is throwing the baby out with the bathwater in my opinion)
• backupd had written 82.42 GB
• corespotlightd had written 51.61 GB

"Fix" Attempts

Feb 11 6:18 PM — trashed ~/Library/Caches

• corespotlightd remained around ≥100 % CPU for 10 minutes
• No indication this did anything to improve my situation

Feb 11 6:28 PM — trashed the contents of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents which immediately resulted in:

• corespotlightd down to <25%
• Disk writing no longer happening constantly
• Every indication this has "fixed" my problems.

This was the 2nd time I've had to "fix" my sluggish, cursor freezing, beach-ball generating 2024 M4 Mac Mini running Sequoia 15.3. I put "fix" in quotes because this is only a temporary solution. The last time I had to implement this "fix" was 3 February, so it seems to last about a week for me. I was no longer keeping Pages documents open unless actively using them ; I think I'll go back to leaving my planner Pages document open like I used to and see how much it cuts down the time before my next "fix".

PS AshkaTheMoltenFury is an hilarious handle.

fronesis47 wrote:

2. I can now report back on my own experiment: I've got a script that runs every 2 or so days and automatically deletes the corespotlight folder. I've now been running for more than a week and (knock on wood) everything is fine. I never notice any issues deleting the folder, and by deleting it every couple of days it usually stays under 2 gigs in size (though I've seen it as high as 5 gigs). In my experience, the problems don't start until the folder gets north of 25 Gbs.

What's odd about these CoreSpotlight folders is how little effect on Spotlight functionality deleting them has. It does seem like for the first ten minutes or so Spotlight will not return results (and stuff like Smart Folders in Mail don't work properly). But after ten or fifteen minutes Spotlight results are fine.

So it seems like macOS is expending significant system resources in terms of both storage space and CPU time, to produce metadata that is of very limited utility.

After intense guru meditation, I believe this is related to version history of the pages document. I didn't know Pages was doing version history, but I'm not surprised, and am grateful, except for this behavior we are all lamenting. Pages->file->revert to->show all versions produces a mighty number of versions, each with small incremental changes (which is desirable behavior, mind). I suspect that some bug is triggering some weird loop therein that spotlight can't contend with. Next stop on the troubleshooting train: exclude ~/Library/Metadata from spotlight. Why would we need spotlight to index that anyway for routine operation?

I've had a similar issue on my M1 Mac Mini, Sequoia 15.4.1 with freeze-ups every couple of minutes lasting 5-10 seconds. The Disk appeared to freeze for this period, unresponsive mouse (but the keyboard would work) followed by a massive surge in write outs and high activity on Kernel_Task and Corespotlightd.

I think I've fixed the issue thanks to several comments on this and other threads pointing towards problems with Pages.

I have recently produces a large document containing HEIC photos transferred to my Downloads folder which I have deleted and then emptied the Trash and the problem has completely gone away.

Thanks to everyone who has helped me and I hope my addition helps someone else.

revpilot wrote:

Thank you Eric. I have gone to that and there is no folder listed as metadata. ** I looking in the wrong place? If I **, I would appreciate you telling me how to find this. I have gone to Settings, clicked Spotlight, Privacy and told it to rebuild, and have not have major success with that. I did this when an Apple Support person told me to this. Any help is greatly appreciated.

The metadata folder I'm referring to is in the user library folder. To get to your user library folder, go to the "Go" menu in the Finder. If you don't see the Library in the "Go" menu, hold down the Option key on the keyboard. Scroll down to the Metadata folder inside the Library folder. Inside, among a few other folders, you will find a folder titled "CoreSpotlight."

Apple recommends that you delete the contents of the CoreSpotlight folder, but not the folder itself. That's how I've always done it. Other users have deleted the folder itself, apparently without ill effects. Either way, macOS will recreate the folder (if you delete that) or its contents (if you just delete the contents). You might need to wait a while for Spotlight searches to return usable results after deleting, so I usually delete the metadata when I know I won't be using the computer for a while, either right before I leave work or before I retire for the evening.

Note that this is not a one-time fix. I typically delete this metadata when it gets above 40 GB, which depending on who knows what other factors can take anywhere from weeks, or months, to less than twelve hours. In my

experience having a large Pages file open reduces the time before I have to delete the metadata again.

I deleted the large folders only in ~/Library/Metadata/CoreSpotlight/. I sorted the folders by size and any folder that has grown in GBs from a previous check gets deleted. My problem happens soon after an update on my M4 iMac. Another indication of the problem is that my GB of used space (System Settings/General/Storage) jumps from what I know to be the used space for me of around 60 GB of space used. I keep an eye on the used space on a regular basis and keep pictures to a minimum.

As this thread approaches its one-year anniversary, there's still no definitive identification of the specific bug, nor any clear admission from Apple that it's a bug. But I do think we've managed to synthesize a lot of information:

• The problem starts with Pages. It's not clear whether the core problem here is the way Pages saves its unlimited revisions, the way it allows for unlimited undos, or the way it builds the index for spotlight, but it starts here.
• Spotlight is clearly involved, as it's building a searchable index, and whatever is wrong with the Pages file gets made worse with spotlight. There is clear evidence for this when the metadata folder balloons to 30, 50, or 150Gb in no time at all.
• Users often only realize there is a problem when the corespotlightd process starts to work overtime, but that's clear a knock-on effect from the size of the metadata folder.
• Cloud activity exacerbates the problem: so moving files out of iCloud and not sharing files may help.
• Deleting the metadata folder is also a very practical workaround.

The big question for me is what is the situation with Tahoe, 26.1?

Multiple posters here have asked about Tahoe, but I get the sense that most of us on this thread with the problem have not updated to Tahoe. I have yet to see multiple reports from users running Tahoe that the problem persists. And that's the only thing that gives me a glimmer of hope...

Update: Slow upload data speeds appear to be causing lags/spinning wheels while working in my larger Pages documents on iCloud. I often work at remote locations using a VPN. I have no control over the upload speeds on these remote Wifi connections (community college Wifi, coffee shops, etc). So, I moved my large Pages files off iCloud drive to my computer drive. Since moving the files, I have not encountered any issues high CPU usage, lags, or spinning wheels while working on those documents. Emboldened, I turned on Apple Intelligence and so far, everything is working perfectly. No processes are hogging the CPU and there are no delays/spinning wheels while working within any app, including Pages.

ericmurphysf wrote:

I definitely don't know, but I think it's safe to assume that this folder (or these folders) definitely should not be more than a hundred GB.

Agreed. All of MacOS only takes up around 23Gb of space. This metadata folder simply shouldn't grow to more than a few Gbs.

I also think it's relevant that there seems to be, on my system, NO DOWNSIDE to deleting the metadata folder. Spotlight works just as well, maybe better than before. (And note that I am NOT reindexing, toggling the indexing status, or doing anything at all with spotlight – just deleting this folder.)

Mitch wrote:

My issues with this process hogging the CPU seem ridiculously random.

But have you ever had the corespotlightd CPU problem when the metadata folder was fewer than 10Gbs in size?

I can attest that I have not. Corespotlightd processes only become a problem for me when the metadata folder grows to around 30Gb or so. Before that, it's fine.

CaptainJoy wrote:

Feb 11 6:28 PM — trashed the contents• of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents which immediately resulted in:

corespotlightd down to <25%

• Disk writing no longer happening constantly

• Every indication this has "fixed" my problems.

This was the 2nd time I've had to "fix" my sluggish, cursor freezing, beach-ball generating 2024 M4 Mac Mini running Sequoia 15.3. I put "fix" in quotes because this is only a temporary solution. The last time I had to implement this "fix" was 3 February, so it seems to last about a week for me. I was no longer keeping Pages documents open unless actively using them ; I think I'll go back to leaving my planner Pages document open like I used to and see how much it cuts down the time before my next "fix".

PS AshkaTheMoltenFury is an hilarious handle.

I'm not certain how significant this is, but I have yet to need to remove the above-referenced folders on either of the two Apple Silicon systems I own: a M1 Ultra Mac Studio or an M2 Max MBP. Neither system's Spotlight metadata folders have ever exceeded 50 GB, and I have not seen serious performance degradations* on either one.

But I've had to remove Spotlight metadata twice each from my two Intel systems, a 27-inch iMac and an iMac Pro. I deleted this data once on each system when it had more than 500 GB of metadata in these folders, and then once again on both systems when metadata exceeded 100 GB a week or so later.

In all four cases, I saw immediate performance improvements, especially in Spotlight searches (which had been essentially inoperative once metadata exceeded about 250 GB), and Time Machine backups never starting let alone completing. And so far literally no downsides whatsoever to removing this data.

_______________________________

• I did for a short time see issues on the Mac Studio with 10-30-second freezes, especially during video playback, one or two of what looked like kernel panics (the system simply shut down entirely the first time, and then shut down and restarted the second time), along with some oddities with Time Machine. Either coincidentally or not, once I removed the excess metadata from the two Intel machines, these problems seemed to resolve on the Mac Studio via unknown and unguessable mechanisms. I should probably note that all four of these systems are under the same iCloud account.