Continued corespotlightd process CPU overload issues

CaptainJoy wrote:

ericmurphysf asked:

How heavy a user of Pages are you?

I use Pages all the time. As a rule, I always have at least one Pages document open, and often one or more are >10 MB.

I suspect this is the root of the problem. I don't know if the amount of metadata accumulated on your system is comparable to mine, but for the last quarter of last year I was editing at least one file (on multiple Macs) that by that time had grown to ~200 MB (roughly 1,200 pages with many embedded graphics), and I'm pretty certain the result was half a terabyte of metadata saved in my user library folder. I strongly suspect a consequence of that amount of metadata was system instability, major issues with Time Machine backups, and degraded search performance in Spotlight to the point of unusability.

Since deleting this metadata last week, all three of these issues have resolved. The one remaining issue is that metadata continues to accumulate at an alarming rate, which will likely force me to remove it just as I did the last time, probably within the next two months. I'm pretty sure this is a bug somewhere in Apple's code that will need to be resolved via a future macOS update.

Hi, I am new Reading this post and writing. I started having these kind of problems a couple of months ago. My MacBook Pro is from June this year, so less than one year old.

I have checket and I also have a store.db file of 17,51 Gb and another one of 1,91 Gb.

What soul I do about it? Can I just delete them?

Thank you ver much!

Currently trying this out to fix that bug myself, and I have absolutely no idea what I'm doing. So I just want to make extra sure I got what you're saying. Now when you said deleting the metadata out of Corespotlight and SpotlightKnowledgeEvents, can I just delete everything inside those folders? Not working with Terminal here. Or keep the folders inside intact and just delete the lists and whatever is in there? Thanks!

Mitch Stone wrote:

I'm not sure what you mean by without migrated data, though. My files and apps were manually migrated on the Studio.

I meant migrated automatically, using the migration assistant. On my M3 air I did not use migration assistant at all, and that machine is not affected by this problem.

Some of the theories for the origin of this issue are interesting but probably have to be treated as theories. As I've mentioned in previous posts, since the last time I deleted the spotlight plist, I have not been afflicted by this issue. It's been a couple of weeks now.

It's interesting that you've had good success with only deleting that plist file. I had previously deleted almost everything, and the problem came back.

I have just now again deleted the entire corespotlight folder in (in the library folder) and the preferences file you mentioned, but I think I see the corespotlight folder already growing again. We'll see.

fronesis47 wrote:

I have just now again deleted the entire corespotlight folder in (in the library folder) and the preferences file you mentioned, but I think I see the corespotlight folder already growing again. We'll see.

This has been my experience too. An hour or so after deleting the contents of these folders, I'll see a few GB of metadata accumulating already. And if I have a Pages file open, that growth will continue. Just a week after deleting all of this metadata (on a computer that had a particular Pages file open and being edited more or less continuously), the CoreSpotlight folders had already grown to over 100 GB.

In my experience, having a Pages file open on a system, at least a large one, but even if you're not actually editing it, seems to accelerate the accumulation of metadata. On one of my systems I had a ~14 MB Pages file open, and metadata seemed to accumulate at 10-20 GB per day. Closing that one file when I wasn't actively editing it greatly slowed down the accumulation of metadata.

As noted elsewhere, my hypothesis is that when you have a Pages file open (especially if you're editing it), the various Spotlight processes don't just reindex the changes; they re-index the entire file, and not by replacing existing metadata but by adding to it. In my case, the Pages files in question are synced via iCloud and are being edited on multiple Macs. It seems that if you close the file, then edit it on another computer, and then later re-open it, Spotlight reindexes the entire file, but only that one time (until you edit it some more). If you leave it open while you're editing it on another system, the same thing happens: Spotlight seems to reindex the entire file with every edit. This can add tens of gigabytes a day of metadata on a file that might only be a few megabytes in size.

Until Apple releases a fix for this issue (which may or may not ever happen), I think the best way to avoid rapid accumulations of metadata is to close Pages files when you're not actively editing them. Even having them open on another system and editing them there (if they're synced via iCloud) will still lead to vast quantities of metadata creation, as much at 20 GB a day.

I agree with Mitch Stone: it would be good to know if other documents (such as Numbers files) cause the same problem, and to see what happens with a new user.

THANK YOU to sugarskyline for showing me how to change the default presentation of the message board.

During all my trials and tribulations with this issue I've had several Numbers files open on various computers (I almost never use Keynote). Most of the Numbers files are not very large (half a megabyte or so), but one is nearly a hundred megabytes. That's mostly embedded graphics though, and I'm not sure how much indexing gets done on graphics.

In any case, Numbers files do not appear to make either corespotlightd or Spotlight metadata spike. YMMV, naturally.

I should probably clarify that whether or not I have multiple Numbers files open on various computers seems not to affect either how much CPU time corespotlightd uses or how quickly Spotlight metadata piles up.

Having even one Pages file open, by contrast, has a huge effect on both.

If the folder is the core problem, what would happen if someone locked it (Get Info > Locked) ? Not well versed on how the OS works so I don't want to attempt in case it breaks something, but theoretically, would be there any value in locking the folder/relevant innermost folder to prevent anything from being written?

sugarskyline wrote:

If the folder is the core problem, what would happen if someone locked it (Get Info > Locked) ? Not well versed on how the OS works so I don't want to attempt in case it breaks something, but theoretically, would be there any value in locking the folder/relevant innermost folder to prevent anything from being written?

Interesting question. I don't know the answer.

I do know that file and folder locking in the current MacOS is a very old legacy of the Classic MacOS, and that the read/write permissions of OS X basically override it.

Of course, you cold also change that folder to read only. I don't know what would happen, but I'm wary of trying it because a bunch of spotlight processes expect to be able to write to that folder. I worry that some stuff could go haywire if you randomly blocked their access.

The good thing about my current workaround is that it minimizes the potential problems, but doesn't break anything.

Mitch Stone wrote:

Earlier in this discussion it was established that iCloud is not the culprit. Files that will trigger the problem will do so whether they are stored locally or in iCloud. I meant corespotlightd because this the process I see as being hyperactive when the CPU is overloaded. Either way I have had this large Pages file open for over an hour now and the file has not grown at all. Unfortunately all of the theories we've come up with so far are incomplete or flawed. They only seem to work for some users some of the time.

I realize iCloud is not the cause per se (because the problem has been reproduced with Pages files that are not on iCloud Drive). I'm curious about exploring the possibility that the problem is worse without the optimize function on. It seems to me that with my two machines, there's just much slower growth of the folder on the machine with optimize on.

I'll also add: I'm not sure your case completely disproves the general thesis. 60Gb is a really large metadata file. It's already at the point that corespotlightd is taking over the CPU. So by this point it doesn't really matter that the folder has stopped (or significantly) slowed its growth; it's already a problem. And I guess I have to assume that the folder got that big for the same reasons as everyone else. If you could delete your folder, and then open a pages file and have the folder not grow – that would be a totally different data point.

Now that I've been deleting my metadata folder (as an experiment) I have absolutely zero issues with corespotlightd. If the corespotlight folder is smaller than 10Gb I find I never have any slowdowns at all.

I have another question: do people with Apple Silicon systems ever see the size of their Spotlight metadata folders decline? I've got two Intel systems and two AS systems, and while I never have seen these folders' sizes shrink (other than when I manually delete them) on Intel systems, I often see them get smaller on their own on AS systems, especially on my M2 Max MBP.

I'm not sure this is actually happening (I suspect it's just the Finder having difficulty giving accurate sizes to folders with a lot of subfolders, which is more the case with AS systems). But I have noted that AS systems do not seem to see as rapid growth of Spotlight metadata as my Intel systems do. I've repeatedly deleted metadata on Intel systems only to see it grow back to an absurd extent. But I've never deleted the metadata out of either AS system, and so far neither one has ever gotten any bigger than 47 MB, and currently neither system has more than 25 GB worth of Spotlight metadata.

ericmurphysf wrote:

I have another question: do people with Apple Silicon systems ever see the size of their Spotlight metadata folders decline? I've got two Intel systems and two AS systems, and while I never have seen these folders' sizes shrink (other than when I manually delete them) on Intel systems, I often see them get smaller on their own on AS systems, especially on my M2 Max MBP.

I can keep an eye out and report back! 🙂