I also own the iMac 5K base model and am experiencing the same problems. I made some benchmarks and decided to post them here for people who read this thread and now are scared that Graphics / the UI is much slower than on normal non-Retina Macs, also, my results seem confirm that it's a software issue. The problems with most benchmarks on the web is, that they don't take in account that the iMac 5K has to push 4 times more pixels than the normal 27" iMac. For example I ran Cinebench (which runs at non-retina resolution on the 5K) and the score was about 4 times higher than on my previous non-retina Mid 2011 21.5" iMac with the AMD Radeon HD 6750M 512MB where I never had any graphic issues while working (I use mostly Emacs, Xcode and a lot of Safari). I decided to use the outdated Xbench for benchmarking because it runs everything in retina resolution when possible and, as far as I can see, runs tests that show much better how the interface may perform.
The benchmark with Xbench showed that compared to the iMac 21.5" Mid 2011 with the AMD Radeon HD 6750M 512MB ...
- Quartz Graphics were about 30% slower
- OpenGL Graphics were a little bit faster
- the "User Interface Test" results were a little bit better
... on the iMac 5K base model. View full results.
This probably means that the iMac 5K can push 4 times more pixels nearly as fast as my previous iMac could push pixels on its non-retina screen. This also complies with the Cinebench results being 4 times faster on the 5K. I'm no benchmarking expert so you may take these results with a grain of salt, also Xbench was updated the last time in 2006 so it may not be using the benefits of Apples latest graphic framework.
PS: Would be great if someone who has Mac with a slower graphics card that really struggles with Mission Control (not sure if that really exists thought) could post his Xbench scores so we see how far the iMac 5K is away from that.