Remote desktop connection extremely unstable when using vlc-player on remote machine - a known issue?

Remote desktop almost unusable when running vlc on remote mac mini trying to control it from mac book pro retina -14.

Also at times extremely slow when running native apple apps - and it's not a network problem.

Seems like it's gotten worse with the latest os x updates. No problem when connecting to mac mini when connecting from a linux-machine. Works flawlessly.

A known issue?