Poor Screen Sharing performance - problem reproduced

I have finally figured out how to reproduce an issue that has recently appeared on a Mavericks Server where some of my users who work remotely via the Screen Sharing app have been complaining about poor screen performance. A number of my users make use of Apple's build in Screen Sharing app to log into their workspaces and work remotely. However, some of them complain of very poor screen performance (lagging curser, slow redraws of windows, etc). I found this puzzling in that each time I logged in to do the same thing, performance seeded good (I chalked it up to "the user mush be on a slow network connection).

However, I finally got the problem to reproduced. Here is what I found: If user X is logged in, and user X's workspace is displayed on the remote machine, user X can work efficiently using Screen Sharing. However, if user Y's workspace is displayed on the remote machine, then user X's experience while working remotely with Screen Sharing is much degraded.

For some reason, the performance of Screen Sharing while working remotely is good only while the remote user's desktop is being displayed on the remote machine. The problem seems to be how graphics resources are allocated (prioritized) by the OS, if high priority being given to the workspace that is displayed on the remote machine.

Does anyone have ideas as to how to fix this? Would running Apple Remote Desktop make any difference?