Audio MIDI Setup constantly freezing after opening

This has always been an issue with macOS but it's unbearable with Mojave.

I can use Network MIDI for a few hours and then suddenly when I'm trying to open Audio MIDI Setup it's completely frozen. Force-closing Audio MIDI Setup, doing

sudo killall coreaudiod

and

sudo kill `ps -ax | grep 'coreaudiod' | grep 'sbin' |awk '{print $1}'`

and deleting

com.apple.audio.AudioMIDISetup.plist

doesn't help at all.

What's worse is that I can't open any Audio/MIDI application like Ableton Live or MIDIMonitor.

After a while the Audio MIDI Setup tells me

The MIDI Server can’t be opened.

Audio MIDI Setup couldn’t open the MIDI Server so you can’t view or edit your MIDI configurations.

If you recently installed a new MIDI driver, try uninstalling it and then open Audio MIDI Setup again. If Audio MIDI Setup works correctly after that, contact the driver’s manufacturer for further support.

But the last time I've installed something like that was YEARS ago.

MIDI Monitor eventually shows a popup saying

Error

There was a problem initializing the MIDI system. To try to fix this, log out and log back in, or restart the computer

Restarting the MacbOok helps but it's annoying to do that every few hours.