Unreliable screen sharing connection to headless MacMini M1 Big Sur

This seems to have fixed itself. I rebooted the server to try to define the problem better. First, I decided I should make sure the TV was turned on when first connecting via Screen Sharing. That worked and the session stayed up for over 3 days. Then I tried to break it. Experimenting with connecting and disconnecting under various conditions, I could get it to fail only once, when it popped up a message saying the server and client were incompatible. I can't remember what I did just before that, but since I was intentionally "abusing the relationship", I feel that's an acceptable outcome.

Embarrassing to think that I may have fixed this with a simple reboot.

My previous reply indication that the issue was solved turns out to be incorrect. The problem returned repeatedly, sometimes just after rebooting. The screen sharing session will fail sometimes within a couple of seconds, sometimes a couple of days.

An Apple senior advisor has assured me that this is a known problem the the M1 graphics driver and a fix should be coming in the next OS release.

Finally called Apple Support and they won't help because I'm now on 11.3 beta, and the only way to revert to 11.2 is to wipe the drive, they say, and reinstall the OS. I don't want to waste time doing that at this time. I'm trying out Duet Air at the moment to see if that will work.

I suspect Apple made an incompatible change in Screen Sharing in Big Sur. Since I was occasionally getting a message saying the client and server were incompatible, I finally bit the bullet and upgraded my client MBP from Catalina to Big Sur. It's only been a couple of days, but I think it's working better so far. I also upgraded my iMac from Mojave to Big Sur.

On Mojave I was having a different problem: instead of disconnecting frequently, the client would stop sending all the keys to the server. For example, CMD-SPACE would stop invoking Launchbar on the Mini and would invoke it on the iMac instead. So far since upgrading the iMac that problem has gone away.

I am still getting occasional disconnects, but I am also stressing things a bit more by sharing the screen from both clients at once. That was not possible before-- the second client just wouldn't connect.

Oddly, the iMac often fails overnight complaining about authentication while the MBP stays connected. And I authenticate from both machines the same way.

Things have improved since I updated my clients to Big Sur, but I'm still having problems that I'm beginning to think are due to limitations of the graphics hardware on the clients. Lower resolutions work better than higher. I use SwitchResX to set the same resolution on both sides so I can take advantage of the maximum resolution of the monitor without scaling. On my 2017 iMac with Radeon Pro 580 8 GB graphics, when using the native resolution of 2560x1440, it works pretty well most of the time, doesn't disconnect permanently, but sometimes has a noticeable lag (all machines are connected by gigabit ethernet) and a couple of times the connection stops and the "Reconnecting..." screen displays for a minute or so, then comes back. But with 2048x1152 resolution (thankfully my preferred setting) it works perfectly and stays up for days.

My late 2013 MacBook Pro has Intel Iris 1536 MB graphics and I use an external Apple Cinema Display for screen sharing (1680x1050 native resolution). When set to that resolution (1680x945 on the Mini) things are pretty much as bad as before. It can disconnect permanently and I have to kill screensharingd on the Mini while actively using it after an hour or two. But 1280x800 (1280x720 on the Mini) resolution seems to work much better. I haven't had time to see how long it stays up at that resolution yet because it's not very useful for my current project--- I need to use the higher res of the iMac. 

Also since my last post I have stopped trying to screen share from both clients at once. There wouldn't be much point since given what I've discovered I would have to run at 1280x720 on both clients.

It does appear that the performance of the client hardware is the key here. I turned off encryption by executing

defaults write com.apple.ScreenSharing encryptRFBDataStream 0

on the server and both clients. This fixed the problem and finally I am able to reliably view screen sharing at full native resolution on the Apple Cinema display.

In hindsight, this is pretty obvious, but when I noticed the problem I'd just started using a machine with a new architecture and new OS, so I assumed the problem was there.

But OS compatibility is also part of the problem. To verify, I tried booting my MBP off a Catalina SSD clone made just before I upgraded it to Big Sur. I turned off RFB encryption and experimented with that for several days, and basically had the same results as before--- frequent disconnects at either resolution.

So is this the place to complain about Apple's consistent disregard for backwards compatibility? I mean, this is VNC, for heavens sake, an ancient protocol. Any client should be able to connect, even an old Windows machine. Why should it only work with exactly the same version of MacOS? I suppose, to give them the benefit of the doubt, that they could have come up with a more efficient transfer protocol in Big Sur, which of course would have to be negotiated and present on both sides, which could add just enough efficiency to overcome the deficiency of the older hardware.

It would be interesting to know if that's the case.

Have you tried a lower screen resolution on the Mini? If you are driving the TV with full 4K res, for example, the mini could be struggling to provide another 4K stream to screensharing. Also, the wifi could be a problem, depending on your router and your wifi configuration options, and competition with other services using wifi. That wasn't an issue in my case, cause everything's connected with gig ethernet.

Good luck!

Well I've updated to 11.2 and then 11.3 beta and the problem is still there.

Thanks for the reply, but I'm not using FileVault