I also had a similar issue. But I am trying to connect my Macbook pro retina display (late 2012 built; running maverick) to an external monitor and use it as a home desktop (so, I don't need to have the MBP screen on! However I include a solution for that as well!). So, after reading articles after articles, and doing everything possible to my monitor and MBP I was able to fix my issue and now I get amazing resolution and picture quality with my ASUS VS238H-P.
So these are the things I did:
Disclaimer: I am not an engineer or a computer geek so what I write here is from a layman terminology, so don't get offended if I'm not completely accurate in my descriptions.
1. Unlike what have been suggested by many people in this thread, you DON'T need to use/buy any cable other that HDMI. If you already have one of those good, if not, don't waste your money. This works with all any connection your laptop has.
2a. If you are like me and don't need to have your laptop open (aka. have a mouse/trackpad and keyboard connected to your laptop), Close the lid of the laptop when you connect your external monitor. This action tells the mac that you are solely using your external monitor and that way it adjusts only to the external monitor's aspect ratio and settings.
2b. If you need to have access to your laptop keyboard/trackpad and want to keep it open, mac OS only gives you the option for "Mirror display". What's annoying about this is that your MBP wants to have it's internal display looking the best at all times, so it stretches/shrinks images and texts accordingly on the other monitor when it adjust it to it's aspect ratio. This is one of many reasons when you connect an external monitor, you get a poor image quality if you choose to optimize the displays for your external monitor. So to fix this issue, go to Display (System Preferences/Displays/Display) and choose "optimized for:" built in display (Ex: in my case, Optimized for: Built-in Retina Display). With this option, your external monitor might have some black edges on the sides (because you are not using it's full aspect ratio) but the texts/images are much more crisp because they don't lose their native aspect ratio (ex:shrink/expand) to fit the external monitors'.
3. I also suggest turning off the "Use LCD font smoothing when available" (System Preferences/General @bottom of the page). This will give you a better text quality when using external monitors.
4. Most importantly, check out these links:
a. http://apple.stackexchange.com/questions/107846/how-to-enable-hidpi-mode-on-exte rnal-display-in-os-x-10-9-mavericks
b. http://www.reddit.com/r/apple/comments/vi9yf/set_your_retina_macbook_pros_resolu tion_to
All the credit goes to these guys! Long story short, the issue is that when you connect a monitor to your MBP, in Displays you only couple of generic options for "Resolution" which are mainly meant for the internal MBP monitors, most of which aren't even close to what you need to have a good picture/text quality on an external monitor. You can increase the number of available resolutions by pressing Option (aka. alt) and then clicking on scaled in (System Preferences/Displays/Display). This was all great news until I stumbled upon the these articles and discovered HiDPI. This is the setting that your retina display uses and by using the correct resolution in HiDPI mode you can make the quality of your external monitor as good as a retina display (that is if your monitor has the capacity which most newer ones do!). So, follow the Reddit article and download the RDM. Then choose an HiDPI resolution that best suits your monitor (ex: I like bigger icons/texts so I'm using 1280 x 720 HiDPI and am loving it!). This will fix all the issues that you will have with bad image quality.
Good luck.
