Frequent system hangs and freezes in macOS Mojave 10.14: Notes, reproducible errors, and possible workarounds

NotUncleAl — Spotlight is a possible culprit, in some circumstances, but it turned out not to be with me.

WindowServer seems to grab a lot of memory in AM, but I'm pretty sure that's expected behavior; it's handing out windowing information to such things as AM, while the window is open, so you'd probably expect to see an increase in its activity as it refreshes its own status.

This Mini is a late 2012, with a 128 GB SSD and a 1 TB rotational disk, originally strapped together in Core Storage as a single fusion drive. Back in July, we moved halfway across the country, and along the way passed over some ludicrously decrepit interstates, with enough cratering to make it seem like they'd suffered artillery barrages. Everything in the back of the truck got jolted around, including the Mini. It wasn't until mid September that I set the Mini up again. At the time, of course, it was still running High Sierra. Then I tried Mojave on my Air. All seemed to go well, so I installed it on the Mini too, and that was when it all fell apart.

Last week I began hand-copying the contents of my Home folder to an external drive, at first using Finder, but the system hangs were making it impossible to proceed. Ultimately I executed a cp -av command in Terminal, and let it grind away. This is a lower-level process and runs in the Terminal session, so contributed a little less to freezing; but periodically I'd see Terminal hang quite a while between one file and the next, and every once in a while it threw file input/output error messages, too. I think in the end I lost about 5% of my data, but that's not bad, considering I hadn't been able to do a Time Machine backup since the Mojave install. (My documents are synced with my Air via the cloud, and virtually all my video, audio, and photo files reside on a media server, which is itself backed up across other disks.)

After wiping the fusion drive, Mojave hung on the install. I'd needed to load from a thumb drive by then; the recovery partition was no longer available, for reasons which will soon become clear. I put High Sierra on another thumb drive and tried installing that instead, and in the process of doing so, reformatted the container which held my fusion drive (oops). That had the effect of splitting the drives apart again, so Core Storage was no longer treating them as a single contiguous volume, but instead an SSD and a rotational drive, contained in the same Mini.

I reformatted them as HFS+ and rebooted back into Mojave's installer, and tried using a new command in terminal:

diskutil resetFusion

What this theoretically does is rebuild your fusion drive into its factory-ship state. It failed for me.

So I formatted the drives again, this time as APFS, and installed Mojave to the SSD, leaving the rotational drive empty. Then, once I got back into my fully-loaded desktop, I started the proctology. I did that because Disk Utility, when it was able to run independently on the 1 TB rotational drive, reported its SMART status as "Failing".

Oh dear.

Disk Utility cannot conduct a low-level format on anything. Even the secure-erase option only whacks the data on a partition; it doesn't pay any attention to empty or deallocated sectors. So there was no inbuilt way, through GUI tools, for me to do a low-level format and check the drive block by block, sector by sector. My next fallback was to get more information on the SMART status, since "failing" is not particularly illuminating.

SMARTReporter, if you don't have it, is something you should get. Even the free version is useful beyond my capacity to easily describe. It exposed more than 31,000 errors on the rotational drive, some of them more than 100 sectors in size, on a disk with less than 47,000 hours total uptime. I'm proximally certain some of those bad sectors once contained the recovery tools, which was why booting to recovery wasn't working any more.

The conclusion was plain: The physical surface of the rotational drive has been damaged, and large parts of it are unreadable now. My first guess is it happened during all the jouncing in the move.

To see how bad the damage was, I tried using this command to write 0 to every block on the disk:

dd if=/dev/zero of=/dev/disk1

…but it finished earlier than it should have, at about 940 GB of 1000 written. This suggests to me there's about 60 GB worth of damage on the rotational drive. (Don't use dd if you don't have a very good idea what you're doing; it'll overwrite anything with anything, even if you didn't want it to, simply following the commands you gave it.)

I might be able to use the command-line third-party program smartctl to build a custom table of blocks to avoid, but at this point, it's likely not worth it. If I tried that, it would just be as an experiment to see if I could.

The point is this: Particularly in fusion drives, it can be hard to spot if there's something suffering from incipient failure, and the odds are very good the Mini was in trouble before I put Mojave on it. MacOS, out of the box, does not actively poll any drive's SMART status. You either have to check using Disk Utility, or you have to get into the About This Mac > System Report… subwindows (look under "SATA/SATA Express"), or you have to install a third-party app to do the checking for you. So I have no way to know for certain when those incipient failures began popping up. I just wasn't paying as close attention to it before the install, because I was not waiting for a new install to finish, load, let me get to work, etc.

The demo/free version of SMARTReporter is here: https://www.corecode.io/smartreporter/

Its diagnostics, under the "Advanced Tools" tab of the "Disk Checks" pane, might disclose information regarding your own drives, as well. The only difference between the free and paid version of SR is the paid version will actively alert and email you if it detects impending failure; it also gives you a nag dialog at boot/login. The paid version is ten bucks.

So the rundown is this: If your Mac is still hanging a week or more after the Mojave install, even if Spotlight is off, run some low-level disk utilities on it and see if one of its drives is on the way to checking out. I don't know if Mojave is more sensitive to this kind of thing, maybe because APFS is finally in the mix across all supported systems, but it certainly does not like disk errors. It will hang, for minutes on end, even if you're not doing anything directly related to disk I/O in either the Finder or inside applications (open/save dialogs).

Hi, sepsus. Happy (?) Monday to you!

PathFinder … I've tried that one myself, and you're right, it does seem to perpetually be in alpha. I settled on TotalFinder a couple of years back, mostly because I wanted full-window labels/tags. It seems to be pretty stable. Of course, that one needs SIP disabled to install or run. It's always something, isn't it?

But even with TF disabled, those hangs persisted.

My own odyssey is far from over, but the Mini seems to have settled down a bit. There was a time when, after a particularly vicious little set of hangs in Safe Mode, it didn't want to boot at all. I couldn't even get Recovery or Internet Recovery to load. While I was downloading the Mojave installer to my Air and creating a bootable installer thumb drive, the Mini managed to bring itself back online, and I resumed copying the content of my home folder.

That specific set of boot problems was preceded by a console error message during restart. A photo of that message is at the end of this post.

Occasional (-36) errors continued to surface, but they remain rare, and are mostly limited to relatively recent files, and so far, only to fairly insignificant and non-system-critical ones. Curiously, the problems I saw before — the whole Mac freezing while Finder attempted to copy the corrupted file — no longer seem to be manifesting.

I've never used Smart Mailboxes in Mail, so their absence isn't conspicuous to me. It wouldn't surprise me to learn they're reliant on Spotlight, though. A few too many things seem to be.

I'm beginning to suspect an interaction (at least in my case) between FileVault, some background file I/O processes, and those few mildly damaged files. Given how I know Mojave behaves when it tries to copy a corrupted file, it's easy to imagine similar symptoms manifesting in background file-access operations.

I have yet to identify a single consistent or persistent cause of the system freezes in Activity Monitor. It's been up and updating since yesterday. It never once showed any indication that any background process had locked up.

However, at the moment, operations on this Mini appear to be smooth and normal.

It's conceivable that, after I freed up 25% of the space on the HD, OSX now has enough room to do what it needs to do with such processes as background defragging, and moving certain files over to (or off of!) the SSD portion of the fusion drive.

FileVault quirks:

The Mini has been running in normal boot for more than twelve hours now, and appears to be considerably more responsive and reactive than it was. It may have been decrypting the boot volume all last night, or it may not. I can't tell.

I can't determine if FileVault has been deactivated or not, or how far along the decryption is. The decryption progress bar is no longer visible in FileVault's window. When I directed the Mini to disable FV, it was still in Safe Mode, and decryption did indeed begin, but a few minutes later, the progress bar simply disappeared. (I turned off FV on my Air too; it took about 2 hours on that machine, and the progress bar was visible the whole time.)

Decryption appears to be a persistent state, something that resumes after system reboot. So when I finally was able to bring OSX back online, the progress bar returned to the FV window, about 2% filled in, with the estimate on the screen telling me there was "about a minute left" to finish the decryption. Not possible; it hadn't been online long enough prior to that, and there are still about 800 MB worth of system and user files on this drive.

Terminal APFS tools show the list of volumes, and indicate the boot volume is still encrypted, and do not indicate it is decrypting. Attempting to run Terminal decryption commands results in an error telling me decryption failed to start. This leads me to speculate FV is still decrypting, but not visibly, not even as a user process in the Activity Monitor window.

The only hint I have that FV's decryption may still be running is the gradually increasing number of bytes read/written in the kernel_task process. But that could also just be normal background file ops, such as optimization.

FileVault is something I've used sporadically. The original version of it created a disk image of your HD, then pretended it was a normal volume after you'd logged in. As a result, you needed to have a lot of free space on your boot volume to activate or deactivate it, so data could be swapped into or out of the disk image. (Once it was running, those space requirements weren't as important.)

The more recent version of FV is a whole-disk encryption-on-the-fly model, so adequate free space to activate it (or deactivate it) is not as important as it once was. However, after it was off on my Air, I noted an increase in free RAM of about 500 GB. That isn't especially surprising — you'd expect the process to use additional system resources — but it was more than I expected.

Summary, so far:

FV, in conjunction with relatively limited (10%) free space at time of install, in addition to post-install Spotlight re-indexing, combined with some damaged files, may have been largely responsible for my initial post-install problems.

I think it would be advisable for anyone attempting a Mojave install to do these things before installing:

1. Disable FileVault.

2. Ensure at least 20% of the destination volume is free space.

…and after install:

3. Disable Spotlight.

4. Disable all system sleep modes (turn off Energy Saver, for instance).

5. Leave the machine running (and, in the case of portables, plugged in) for at least a day.

Somewhere during that time, the machine should be able to get itself more or less functional. After it appears to be working normally, re-activating Spotlight might be viable, though I think I'd still turn it off for items such as spreadsheets, PDF's, and so on, unless text search of individual documents' content is important to a given user.

Historically, Spotlight has sometimes been a problematic service on OSX. We might be seeing that again with Mojave. It's usually fixed with the next release, so 14.1 may bring us a better-behaved Spotlight.

Things I have yet to do:

1. Finish copying my home folder.

2. Complete a TM backup.

3. Reboot.

It's possible this isn't over yet, sigh.

Error message:

Last night, while in Safe Mode, I directed the Mini to disable FileVault. It began to do so, but FV decryption progress disappeared from the FV tab in the Security preference pane. After some time (and a few freezes), I rebooted the Mini. I boot in verbose mode, and saw this final shutdown message (the one beginning "sanity_check"; the "apfs_stop_bg_work" messages appear to be standard fare with APFS).

The system appeared to lock up, and never powered down. A hard restart resulted in a normal flow of console messages at first, then a brief flash of the Apple logo (white on black) with a progress bar, but then a black screen for many minutes.

Thereafter, the Mini would boot in neither Recovery nor Internet Recovery mode. However, as I was creating a bootable Mojave thumb drive, the system eventually pulled up its socks and presented me with a normal login prompt and, ultimately, my desktop.

FV decryption reported it was continuing, but its progress bar has since, once again, vanished from the FV tab.

I have not rebooted since, am still continuing to manually copy my home folder to an external volume, have encountered a few more (-36) files, and system hangs appear to have largely stopped now.

Here's what I can suggest for now, sepsus. (This is based partly on the fact my Mini is now responsive enough for me to use it, most of the time; it's what I'm posting here from right now.)

1. Leave Spotlight off.

2. Log in to your Mac, disable all system sleep functions (Energy Saver, etc), and let it sit, powered up and idling, for a day or two. (Obviously, this won't be palatable if it's your only machine.)

3. See if it's become less laggy, or appears to be functioning normally.

If it is, try reactivating Spotlight system-wide. You may have to let it sit a while again afterward.

With a new system install, a lot of files are copied or overwritten. The new system will start housecleaning right away, and that includes moving pre-existing data from one location to another. This would normally, probably, only take a few hours on a rotational hard drive, a few minutes on an SSD.

But if Mojave is detecting any errors in any files, I think it goes through a couple of steps before giving up on them. I think it tries to repair apparent corruption first, and it might be looking all over the drive to find loose bits that seem to belong to it. While it's doing that, it forget about everything else.

So it's possible that letting the machine do its thing on its own will result in a cleaned-up filesystem and few to no subsequent hangs. It's just getting there that's the problem.

After that, once the computer is more or less happy again, reactivating Spotlight should work. There'll probably be more lagging as the index is rebuilt, but ideally it'll be building an index of cleaned-up and defragmented files, so shouldn't turn into a lethal cycle of system hangs.

For now, I've disabled Spotlight systemwide. The most reliable and direct way to do that is to follow these steps:

1. Open System Preferences.

2. Click Spotlight.

3. Click the "Privacy" tab.

4. Add your entire hard drive (i.e., "Macintosh HD", or whatever you've named it). You can do that by (a) dragging its icon to the "Privacy" list, or (b) clicking the "+" button, then selecting it from the choose-folder dialog box.

I've done this both on the problematic Mini, and on my Air, which is a 2014 model and has not experienced any hangs. Spotlight appears to be a runaway process (see the image below to see what I mean), and SSD's have a finite read/write cycle life. Spotlight's incessant running and indexing of my drives will adversely affect their operational life, and there is no amount of file indexing which I think supersedes the operational life of my machine.

Furthermore, at the moment, Spotlight is hitting my Air's SSD so much that its battery life has dropped by at least 50%. I installed Mojave on the Air several days ago. It's only a 512 GB drive, and about 175 GB are free. There is no way it should take several days to completely index that drive.

The above is Activity Monitor's list of all processes called "mdworker" on the problematic Mini. mdworker is a Spotlight process; its task is to trawl your drive, find searchable content, and cache it.

There are 34 instances of mdworker listed here. None of them have fewer than 3 threads open. That is far, far, far too many. Spotlight is a runaway process, at least on the Mini, and I don't consider it mission-critical enough to let it trash my Air, either.

Meanwhile, the file I/O problem (copy errors on the Mini) may be the other culprit on that machine, responsible for the system hangs. Every time I copy a corrupt file, the system hangs while it tries to read it. And what do we imagine happens whenever mdworker opens a corrupt file to try to read it for Spotlight indexing, or OSX tries to move the file for defragging/optimization?

Gabriel (Andreaux) — thanks. I know Apple does browse these fora, at least, so I'm sure they're aware both of problems, and of various solutions, and of what didn't work out at all. I filed a bug with Apple Developer, too, on the file-copy hangs, but since then have learned a few more things about my specific issue. I'll post the full details in my next reply here, to avoid duplication, since some of it may be relevant to NotUncleAl, as well.

There do seem to be some issues with Spotlight going wild in this release. It's happened before. You jump into Activity Monitor after a full OS upgrade, and see two dozen or more instances of mdworker chugging away. Spotlight could do with some minor addressing there, maybe limiting itself to ten or a dozen spawns, to prevent exactly this thing from happening.

I'd also like to see it assess the machine architecture with a bit of intelligence, maybe launching fewer instances of mdworker when the drive is rotational or fusion, maybe more instances when it's SSD. All those mdworker instances are trying to read data, and that takes longer on rotational (or fusion, which is at least half rotational) than it does on SSD, due to simple mechanical limitations.

I have now found two ways to consistently bog and freeze my Mac. My followup here discusses what I've been seeing, and proposed user solutions. It also goes into what I think the underlying cause is.

Laggy, frozen, slow Mojave: 100% consistent methods to reproduce, discussion of possible cause, and proposed user workarounds