X11 issues when OSX is the server role.

For those following along at home, only X apps are available via the XQuartz X window package. XQuartz provides an X Window Server for macOS, which allows X apps to display locally. macOS itself does not use X for its app displays. Contrast this with most Linux and Unix systems, which use X for local app GUI displays, as well as remote app GUI displays using X. macOS does not use X, and macOS and apps do not use X as its display manager. XQuartz adds this, and only for X apps.

In what you are referring to as case 2, the X server is on Linux. Case 2 is macOS running as an X client, not as an X server. In most (but not all) cases, the X server is running local to the display.

When you direct your local X connections to another X Window Server host through ssh, the preferred outcome is either some obscure X error usually involving authentication or X cookies, or an app display on the specified target X Window Server system.

If you use ssh -Y or ssh -Y or ssh -X -Y to connect into a host, the X apps on that host are then redirected through that path, as the target system gets the session’s default X display redirected and the ports forwarded back to the host originating the ssh connection.

If you ssh -X or ssh -Y or ssh -X -Y from Linux to macOS, then the displays of the X apps—and only the X apps—should be port-forwarded and redirected back to Linux and the X Window Server here. And in this case, the ssh session does not have access to the GUI, as you’re not logged into the macOS GUI. And as the Mac GUI is not X, you won’t see that anywhere else anyway.

So your issue here is that some X app launched in an ssh background session (and not an X app launched in the GUI!) on a macOS system running XQuartz is displayed locally on macOS, and is not port-forwarded to Linux? That sounds like a bug, and maybe a minor security bug, as I’d not expect an ssh session to be able to reach the GUI, which implies XQuartz is allowing that background login access to the GUI.

Open a ticket on github: https://www.xquartz.org/Bug-Reporting.html

And again, this has nothing to do with macOS Server product, this is all about Mac client, and XQuartz.

You state “issues”, give two very involved examples of something using X via ssh, but not indicate what those issues are.

macOS does not use X for its graphics, which’ll constrain some of what can be displayed.

For those unfamiliar with X, the host running the X display is the X server. X clients connect to the X server to display X output.

OS X Server isn’t related to X, it is a package that provides an MDM. OS X Server was formerly a package that provided many useful network services, including Mail and Jabber, among other features. You’re effectively using macOS Client, here. Not Server.