Continued corespotlightd process CPU overload issues

fronesis47 wrote:

2. I can now report back on my own experiment: I've got a script that runs every 2 or so days and automatically deletes the corespotlight folder. I've now been running for more than a week and (knock on wood) everything is fine. I never notice any issues deleting the folder, and by deleting it every couple of days it usually stays under 2 gigs in size (though I've seen it as high as 5 gigs). In my experience, the problems don't start until the folder gets north of 25 Gbs.

What's odd about these CoreSpotlight folders is how little effect on Spotlight functionality deleting them has. It does seem like for the first ten minutes or so Spotlight will not return results (and stuff like Smart Folders in Mail don't work properly). But after ten or fifteen minutes Spotlight results are fine.

So it seems like macOS is expending significant system resources in terms of both storage space and CPU time, to produce metadata that is of very limited utility.

I have iMac 27" intel. I delete System Data related to Library>Metadata>"CoreSpotlight" and "Spotlight KnowledgeEvents" EVERY NIGHT (and delete from Trash, too), or I hit the 50GB threshold that affects pace of storage accumulation and performance significantly. Thanks to this Group for this bandage to keep operational.

My tech knowledge is maxed out at starting my lawnmower. Amazed that I found this group. Apple Support let me down badly. At year end, I began to get pop-ups about "Your system has run out of application memory." Apple support baffled. Ultimately they had me delete and refill MacOS and all data, which is still causing me grief. All for nothing. Sigh...

After intense guru meditation, I believe this is related to version history of the pages document. I didn't know Pages was doing version history, but I'm not surprised, and am grateful, except for this behavior we are all lamenting. Pages->file->revert to->show all versions produces a mighty number of versions, each with small incremental changes (which is desirable behavior, mind). I suspect that some bug is triggering some weird loop therein that spotlight can't contend with. Next stop on the troubleshooting train: exclude ~/Library/Metadata from spotlight. Why would we need spotlight to index that anyway for routine operation?

Could you please define large in "large Pages document"?

I had trouble with Pages files 3-6 MB. Since then, I deleted the metadata several time, narrowed Spotlight's scope, eventually to zero, turned off AI, and even switched to Word for a while. Obviously rebooted. Finally the CPU settled down. Now, I have turned Spotlight back on, turned on AI, and gone back to using Pages with the same 3+ MB files, so far no CPU problems. Magic!? (The System data is ridiculously large at 100 GB, but that is not a problem compared with the lag from an over-busy CPU.)

Update - yeah - the mouse update changed functioning only when Pages is not open. Still having lag issues as well as CPU and CoreSpotlight Metadata problems.

For me Pages is a vital tool that cannot be replaced. Dumping the corespotlight metadata one a week or so will have to do until they can do the fix.

REALLY HOPING they are not planning to ditch Pages… it is the most usable program outside of InDesign for my purposes - and InDesign is WAY too expensive!

Mitch Stone wrote:

Just to be clear, everything we suggest here can be nothing more than a Band-Aid. Only Apple can provide the fix. We are told Apple engineering is monitoring this discussion, so any real-world experiences we can document, and anything that works even as a temporary solution, not only helps us manage the problem in the short term, but perhaps will also point Apple towards the permanent solution we all want.

All this said, based on the now 12(!) pages of discussion since I started this thread, I have become convinced that the problem is Spotlight trying to index documents with a large number of edits. This is exactly how it manifested for me, with an 80k word Pages document being edited by two people with Track Changes turned on. Between us, this resulted in probably more than a thousand edits. Towards the end of the editing, I was seeing beach balling every time I opened this document for more than a few minutes at a time, and had one kernel panic.

Once this editing process was completed, I Finder copied the document. I can now open and make additional edits to the copy without incident. If I watch Activity Monitor (I leave it open in Stage Manager with corespotlightd selected), I will see some spikes in the process, but they are not nearly as high or as prolonged as before, and I also don't see any beach balls. To me, this proves the theory pretty conclusively.

Just to add to Mitch's experiences: I am an inveterate journal-keeper, writing in Pages on a daily basis. It needn't be emphasized that a daily journal grows over time, and by the last couple of months of the year, my Pages journal file is typically over a thousand pages, 750k words or more, which with dozens of embedded graphics will result in a file size between 200–300 MB. Moreover, after nearly a year, we're talking literally tens of thousands of edits. When I was first researching this issue at the beginning of this year, I discovered that my Spotlight metadata had increased on two Intel systems to over 500 GB each. On both machines, Time Machine had essentially ground to a halt, Spotlight search results were useless, both systems suffered multiple kernel panics, and corespotlightd would sometimes pin all CPU cores on both systems, using as much as 1400% of CPU time spread across all sixteen cores (eight of which are virtual hyperthreading cores).

For me, deleting Corespotlight metadata resolved all of these issues. Not permanently; I have to weed out the metadata folders every week to ten days. But so long as I keep that metadata below ~50 GB (on relatively high-performance systems with lots of storage space), my computer life remains pretty peaceful. So long as I remember to quit Pages when I'm not actually using it.

But that said, I was editing my journal file last night, and with that one file open in Pages, which is about 18.5 MB, I watched Spotlight metadata grow by literally a megabyte per second or more, for as long as the file was open. In a single day that would add thousands if not tens of thousands of MB of metadata. But as soon as I quit Pages, metadata growth stopped in its tracks, and was the same value this morning before I left for work, twelve hours later.

YMMV, obviously, but for me these kinds of results could not be more dispositive of the problem here: Spotlight coupled with Pages files with many edits.

"A "clean" install of macOS. First of all, no such process really exists."

Agreed in principle. Apple does offer Disk Utility on its recovery tool to erase the entire data drive, or the entire hard drive, and use internet recovery to restore the OS that came with the Mac, or in recovery, the last installed OS. Simple recovery keeps a partition with just the necessary tools to install the OS.

Use macOS Recovery on an Intel-based Mac - Apple Support - command-option-shift-R for Intel Macs.

Use macOS Recovery on a Mac with Apple silicon - Apple Support (these are Macs that were new starting November 2020 and later)

Either will give you a closer approximation of whether or not the issue is due to the storage issues, or issues with the computer. The Apple Hardware Diagnostics if they find something wrong could indicate something is wrong with hardware.

Just to provide an update and add more context to the discussion:

I'm no longer experiencing any hiccups, mini-freezes, or beachballing. While I’m currently writing my thesis in Pages, my issue might not have been directly related to the app after all.

A few key points:

• I'm working entirely with iCloud files (Pages), and while corespotlightd still spikes occasionally, it does so as expected and without causing any slowdowns.
• kernel_task and corespotlightd are no longer consuming excessive CPU resources.

The turning point seemed to be when I deleted my 50GB ~/Library/Caches folder. After restarting, the system automatically rebuilt the Cache folder, but it has remained steady at around 600MB ever since.

Additionally, I reset my Spotlight index, first via System Settings and then using Terminal with sudo commands to wipe and rebuild it. After indexing completed (which took around an hour or so), my ~/Library/Metadata/Corespotlight folder settled at around 30GB. Despite its size, my system is now running as smoothly and responsively as I expect it to be.

I personally can't work without spotlight anymore and try to index most of the files on my system to have an extremely fast access via CMD+Space. What I'm trying to say is: although my Spotlight folder currently seems to be this big I'm not seeing any performance issues on my end.

Things too keep in mind about my situation:

• I'm not working on super big Pages files with file sizes over dozens of MBs
• I'm not using storage optimization via "Store in iCloud" in the storage settings
• I have updated from 15.2 to 15.3 with obviously very big Cache and Indexes (~/Library/Caches & ~/Library/Metadata/Corespotlight basically never emptied since I own this M1 MacBook from 2021 onwards)

The procedure for deleting the appropriate metadata can be found here: Continued corespotlightd process CPU over… - Apple Community. If that link doesn’t work, you’re looking for a post that ericmurphysf wrote Jan 28, 2025 9:46 AM in response to Mitch Stone.

Mitch Stone wrote:

Maybe off-topic, but perhaps helpful for your Time Machine issue: TM maintains a library of backup images on the backed up volume, so it might not address an issue with a corrupted TM backup by starting with a new backup drive. You might try forcing TM to verify the existing backup. Control-click on the backup drive in System Settings/General/Time Machine and select "verify" from the popup menu. If it can't be verified you will have the option to delete and backup from scratch. Worth a try.

I should clarify what I did which did not resolve the Time Machine situation on one of the two Intel Macs:

• Remove both backup drives from the backup rotation. This effectively disables Time Machine entirely since it has no drives to back up to.
• Erase both backup drives via Disk Utility (APFS encrypted case insensitive in case that matters; I know Time Machine has a hard time backing up to HFS+ drives and has for a while).
• Add the freshly-erased drives to the backup rotation.

In my experience, starting off in this condition (no existing Time Machine backups, freshly-erased drives), the initial backup should start immediately and back up a drive with say 650 GB of data on it in two or three hours. But since I first encountered this issue (endlessly "preparing" backups, kernel panics during or shortly after a backup), that has not been the case. Even an initial backup will be "preparing" for hours or even days, and even if it eventually finishes the initial backup the next backup will take even longer to "prepare."

This suggests to me that the problem is at least partially external to backupd, and given what I understand about the interactions between Time Machine and corespotlighd I suspect that the two are contending for system resources in some fashion. In any case, the benefits of backups to these two systems are outweighed by the inconvenience and hazard of repeated kernel panics.

Mitch Stone wrote:

Having tried this myself, I must report a non-confirmation. I opened a large Pages file and watched the Corespotlight folder file size. It started out at 60.35 GB and remained exactly this size after a half hour, even though the process showed as being very active (100+ percent) for part of this time. I don't doubt that deleting it has a temporary effect but it's also clear that this folder growing in size cannot be triggered predictably by opening a Pages file.

This is helpful. Some key questions:

1. When you say "the process showed as being very active" do you mean corespotlightd? For the record, when I repeatedly watch the corespotlight folder grow (with a Pages file open), it is NOT associated with the corespotlightd process. To the contrary, it's mdworker and mdstores that are writing all the data to that folder.
2. Do you have optimize iCloud storage turned ON, on your machine?

On my Mac with optimize OFF, the corespotlight folder always grows with a Pages file open. But on my Mac without optimize storage ON, I do not see the growth as consistently.

CaptainJoy wrote:

Feb 11 6:28 PM — trashed the contents• of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents which immediately resulted in:

corespotlightd down to <25%

• Disk writing no longer happening constantly

• Every indication this has "fixed" my problems.

This was the 2nd time I've had to "fix" my sluggish, cursor freezing, beach-ball generating 2024 M4 Mac Mini running Sequoia 15.3. I put "fix" in quotes because this is only a temporary solution. The last time I had to implement this "fix" was 3 February, so it seems to last about a week for me. I was no longer keeping Pages documents open unless actively using them ; I think I'll go back to leaving my planner Pages document open like I used to and see how much it cuts down the time before my next "fix".

PS AshkaTheMoltenFury is an hilarious handle.

I'm not certain how significant this is, but I have yet to need to remove the above-referenced folders on either of the two Apple Silicon systems I own: a M1 Ultra Mac Studio or an M2 Max MBP. Neither system's Spotlight metadata folders have ever exceeded 50 GB, and I have not seen serious performance degradations* on either one.

But I've had to remove Spotlight metadata twice each from my two Intel systems, a 27-inch iMac and an iMac Pro. I deleted this data once on each system when it had more than 500 GB of metadata in these folders, and then once again on both systems when metadata exceeded 100 GB a week or so later.

In all four cases, I saw immediate performance improvements, especially in Spotlight searches (which had been essentially inoperative once metadata exceeded about 250 GB), and Time Machine backups never starting let alone completing. And so far literally no downsides whatsoever to removing this data.

_______________________________

• I did for a short time see issues on the Mac Studio with 10-30-second freezes, especially during video playback, one or two of what looked like kernel panics (the system simply shut down entirely the first time, and then shut down and restarted the second time), along with some oddities with Time Machine. Either coincidentally or not, once I removed the excess metadata from the two Intel machines, these problems seemed to resolve on the Mac Studio via unknown and unguessable mechanisms. I should probably note that all four of these systems are under the same iCloud account.

Glad I found this thread and hope everyone's efforts here will lead to a real fix in the next macOS update.

On my side: I use Pages extensively, albeit for relatively small files. I noticed the Pages app was taking 4 Gb of RAM, which is quite surprising. corespotlightd was indeed through the roof on CPU usage. This was heavily draining the battery, I was starting to get disappointed by Apple Silicon chip supposedly being superior here. I suspect things like that have been happening for a while but didn't find out why.

I am using:

• M3 Pro Macbook Pro
• 18 Gb RAM
• macOS 15.3.1

I have been noticing my hard drive is taking tens of Gb of space I am not aware of, in a few days. The folder Library/Metadata/Corespotlight is 40 Gb. I haven't touched it yet.

I disabled sharing Spotlight data with Apple.

Thanks everyone and I hope we'll get to the solution!