Level 1

72 points

Frequent system hangs and freezes in macOS Mojave 10.14: Notes, reproducible errors, and possible workarounds

Like (too) many others, I’ve discovered Mojave to be a source of exasperation, particularly because it seems to lock up periodically, sometimes for minutes at a time, with no identifiable cause.

I’ve been closely watching this behavior, with the result that I’ve been able to reliably reproduce a system hang, make some surmises regarding the cause — at least in my case — and come up with a few suggestions that might work around the problem. Some of these suggestions are for users. Some are suggestions for Apple.

System install on Mac Mini (late 2012), 1TB rotational + 128GB SSD fusion drive. ~120GB free space at time of install. 16 GB RAM.

Symptoms:

• Beachballing. All active apps lock up and beachball. This happens whether there’s a lot of RAM available (50%+), or very little (a few hundred MB).

• Switching to different open app windows works, but any open window remains nonresponsive until the system hang clears. (I do not run anything in fullscreen mode, so I have no idea if task switching in fullscreen works or not.)

• Windows are draggable and refresh as expected while being dragged, but none of their contents respond to user commands (scrollbars are nonresponsive, clicking selectable items results in no state change, etc.) until after the hang clears.

• Switching to another desktop rarely works. If it does, the system remains unresponsive on the new desktop until the hang clears.

• Switching to Mission Control does not work.

• Any new app will not launch until the hang is cleared.

• The tags which typically appear over hovered items in the Dock are slow to respond, or nonexistent.

• Clicking a stack in the Dock doesn’t result in the stack opening.

• Ctrl/right-clicking results in no popup menus, even where they’re expected.

• There is no indication in Activity Monitor of any background process that has run away with the processors, or swamped memory.

• Activity Monitor only reports the beachballed apps as “Not responding”. Force-quitting or getting system info on them is not possible; Activity Monitor’s capability to respond to user commands is just as hosed as everything else.

• User commands are queued, and acted on after the hang clears; so normal commands to switch desktops (as an example) execute in series after the hang is cleared.

• Copy operations on large files sometimes bogs, then freezes, and throws (-36) errors. This was, for me, a significant clue.

Attempted fixes/workarounds:

1. Rebuilding file/folder permissions in Home folder, and at the root volume level.

2. Booting to safe mode.

3. Flushing caches.

4. Resetting PRAM.

None of these had any corrective effect.

Beachballing still occurred in safe mode, but its frequency was noticeably reduced. There were no instances of mdworker running in Activity Monitor while in safe mode.

One thing that did help was disabling Spotlight indexing for the entire volume. This did not correct the system hangs, but it made them less frequent.

Reproducible error:

Large files, which existed on this volume prior to the install, ground to a halt when being copied from the internal HD to an external USB-connected hard drive, and to networked volumes. I have about 110 GB of ripped DVD files originally intended for my media server, which could be copied and purged to free up space. Mostly, these files copied without incident, but occasionally, the copy operation would slow to a crawl, then grind to a halt for anywhere from 15 seconds to a minute or more.

While that happened, all open apps began beachballing.

Eventually Finder would throw an error:

“The Finder can’t complete the operation because some data in ‘the name of the file being copied’ can’t be read or written. (Error code -36)” [OK]

Clicking OK cleared the error dialog, but it was usually upwards of half a minute before the beachballing stopped. Thereafter I could locate the file in Finder, select it, and trash it.

Initially, I just tried re-copying the file, but Finder consistently bogged, halted, and threw the (-36), at the same number of reported MB copied, each time. Clearly the files in question do, indeed, have problems, and something involved in file I/O cannot work past them.

Relevant observations:

• OSX defragments in the background.

• Spotlight goes read/write crazy on any new system install, entirely rebuilding its index.

• Most of the fusion drive is rotational media, and therefore slower, relative to SSD.

• Most of the fusion drive is occupied by data.

Surmise:

Some particularly large files might be nominally incomplete or mildly corrupted. This incompleteness or minor corruption might be unnoticeable to the user or to most apps, but some process in Finder (or APFS) may consider the corruption irreconcilable.

These files may have fragments scattered, in bits and pieces, all over the drive.

It is conceivable these files became corrupted during normal system defragmenting; when I first encountered these system hangs, I toggled the power on more than one occasion to force a reboot.

Non-user-facing processes such as Spotlight indexing or file defragmenting might be encountering these files, and discovering the same errors that surface during file-copy operations, with the result being that the system hangs without any user-visible or user-initiated cause.

These symptoms may manifest on SSD volumes, as well. SSD is much faster than rotational media, but all that would mean is the duration of system hangs would be reduced. They’d still happen.

If my surmise is correct and global system hangs are all down to failed file I/O, creating a new user account would not correct the problem, because the corrupted data would still be present on the drive, and OSX would still be attempting to index it, or defrag it, or both. The new user account wouldn’t see or own any of that data, but the machine would still be working on it in the background, as part of normal global system processes.

Possible user steps to ameliorate (untested as of now; I’m still stuck in [-36] purgatory while trying to copy and trash):

1. Increase free space on the volume to 20% or more.

2. Disable Spotlight indexing of large files (e.g., photos or movies).

3. Disable system sleep and let OSX churn through the data over the course of multiple days/nights.

4. Copy large, relatively unimportant files to an external volume, and remove them from the Mini. (Currently in progress; this was how I discovered the reproducible error.)

Suggestions for Apple:

1. Robustify error-handling in file I/O operations. Finder (or APFS) should not grind to a halt, and bog the entire rest of the machine, when data appears to be missing or damaged during file I/O.

2. Analyze Spotlight indexing relative to background defragmenting, and do not allow both processes to be running at the same time, particularly when there isn’t much free space on the drive. Prioritize defragmenting over Spotlight indexing.

3. Make mdworker and Spotlight a little smarter about disk usage. When a volume is at 90% capacity, Spotlight should not be as prominent a process, and should not be actively reading huge installments of data, then writing out extensive cache files. Free space is far more precious than file indexing, on a largely-full volume. Either have Spotlight limit itself to only a few processes at a time in this case, or provide users with a “slim” option that lets us search by filename, but not content. In fact:

4. Give users an option in Spotlight so it only builds an index of filenames, and does not analyze documents for metadata or any other searchable content at all. When I’m searching for a document, I’m searching by name. I don’t need or want to see a list of every text or Web file on my drive that contains every word in my search parameters. This might be useful in Mail, but it is not useful in Finder. Spotlight does not need to be an exhaustive grep tool with a GUI front end. If I want grep, I have Terminal.

5. Allow error queuing for user-initiated file-copy operations. I should not have to respond to a modal dialog, and re-initiate copy, when one document of 500 or so throws a (-36) error. I’d rather the system kept a running tally of what failed, continue the copying with the next file, and present me with a list of failed files at the end of the entire operation.

macOS Mojave (10.14)

Posted on Oct 6, 2018 10:51 AM

Top-ranking reply

WarrenO Author

Level 1

72 points

Posted on Oct 19, 2018 11:16 AM

NotUncleAl — Spotlight is a possible culprit, in some circumstances, but it turned out not to be with me.

WindowServer seems to grab a lot of memory in AM, but I'm pretty sure that's expected behavior; it's handing out windowing information to such things as AM, while the window is open, so you'd probably expect to see an increase in its activity as it refreshes its own status.

This Mini is a late 2012, with a 128 GB SSD and a 1 TB rotational disk, originally strapped together in Core Storage as a single fusion drive. Back in July, we moved halfway across the country, and along the way passed over some ludicrously decrepit interstates, with enough cratering to make it seem like they'd suffered artillery barrages. Everything in the back of the truck got jolted around, including the Mini. It wasn't until mid September that I set the Mini up again. At the time, of course, it was still running High Sierra. Then I tried Mojave on my Air. All seemed to go well, so I installed it on the Mini too, and that was when it all fell apart.

Last week I began hand-copying the contents of my Home folder to an external drive, at first using Finder, but the system hangs were making it impossible to proceed. Ultimately I executed a cp -av command in Terminal, and let it grind away. This is a lower-level process and runs in the Terminal session, so contributed a little less to freezing; but periodically I'd see Terminal hang quite a while between one file and the next, and every once in a while it threw file input/output error messages, too. I think in the end I lost about 5% of my data, but that's not bad, considering I hadn't been able to do a Time Machine backup since the Mojave install. (My documents are synced with my Air via the cloud, and virtually all my video, audio, and photo files reside on a media server, which is itself backed up across other disks.)

After wiping the fusion drive, Mojave hung on the install. I'd needed to load from a thumb drive by then; the recovery partition was no longer available, for reasons which will soon become clear. I put High Sierra on another thumb drive and tried installing that instead, and in the process of doing so, reformatted the container which held my fusion drive (oops). That had the effect of splitting the drives apart again, so Core Storage was no longer treating them as a single contiguous volume, but instead an SSD and a rotational drive, contained in the same Mini.

I reformatted them as HFS+ and rebooted back into Mojave's installer, and tried using a new command in terminal:

diskutil resetFusion

What this theoretically does is rebuild your fusion drive into its factory-ship state. It failed for me.

So I formatted the drives again, this time as APFS, and installed Mojave to the SSD, leaving the rotational drive empty. Then, once I got back into my fully-loaded desktop, I started the proctology. I did that because Disk Utility, when it was able to run independently on the 1 TB rotational drive, reported its SMART status as "Failing".

Oh dear.

Disk Utility cannot conduct a low-level format on anything. Even the secure-erase option only whacks the data on a partition; it doesn't pay any attention to empty or deallocated sectors. So there was no inbuilt way, through GUI tools, for me to do a low-level format and check the drive block by block, sector by sector. My next fallback was to get more information on the SMART status, since "failing" is not particularly illuminating.

SMARTReporter, if you don't have it, is something you should get. Even the free version is useful beyond my capacity to easily describe. It exposed more than 31,000 errors on the rotational drive, some of them more than 100 sectors in size, on a disk with less than 47,000 hours total uptime. I'm proximally certain some of those bad sectors once contained the recovery tools, which was why booting to recovery wasn't working any more.

The conclusion was plain: The physical surface of the rotational drive has been damaged, and large parts of it are unreadable now. My first guess is it happened during all the jouncing in the move.

To see how bad the damage was, I tried using this command to write 0 to every block on the disk:

dd if=/dev/zero of=/dev/disk1

…but it finished earlier than it should have, at about 940 GB of 1000 written. This suggests to me there's about 60 GB worth of damage on the rotational drive. (Don't use dd if you don't have a very good idea what you're doing; it'll overwrite anything with anything, even if you didn't want it to, simply following the commands you gave it.)

I might be able to use the command-line third-party program smartctl to build a custom table of blocks to avoid, but at this point, it's likely not worth it. If I tried that, it would just be as an experiment to see if I could.

The point is this: Particularly in fusion drives, it can be hard to spot if there's something suffering from incipient failure, and the odds are very good the Mini was in trouble before I put Mojave on it. MacOS, out of the box, does not actively poll any drive's SMART status. You either have to check using Disk Utility, or you have to get into the About This Mac > System Report… subwindows (look under "SATA/SATA Express"), or you have to install a third-party app to do the checking for you. So I have no way to know for certain when those incipient failures began popping up. I just wasn't paying as close attention to it before the install, because I was not waiting for a new install to finish, load, let me get to work, etc.

The demo/free version of SMARTReporter is here: https://www.corecode.io/smartreporter/

Its diagnostics, under the "Advanced Tools" tab of the "Disk Checks" pane, might disclose information regarding your own drives, as well. The only difference between the free and paid version of SR is the paid version will actively alert and email you if it detects impending failure; it also gives you a nag dialog at boot/login. The paid version is ten bucks.

So the rundown is this: If your Mac is still hanging a week or more after the Mojave install, even if Spotlight is off, run some low-level disk utilities on it and see if one of its drives is on the way to checking out. I don't know if Mojave is more sensitive to this kind of thing, maybe because APFS is finally in the mix across all supported systems, but it certainly does not like disk errors. It will hang, for minutes on end, even if you're not doing anything directly related to disk I/O in either the Finder or inside applications (open/save dialogs).

View in context

Frequent system hangs and freezes in macOS Mojave 10.14: Notes, reproducible errors, and possible workarounds

Similar questions