Thanks, Malcolm. Yes, you are right. I actually have a Mac Pro
Yes, when I try the first DVI port, the 27" in not supported -- I only get the two lower res options. WHen I plug into the other port, the 27" is recognized and I get all my res options but the weird color artifacting thing happens.
It's probably is the graphics card. That model tends to fail after a few years.
Try a Safe Boot:
That turns off hardware graphics acceleration. If that fixes the colors then the problem is almost certainly the graphics card. (Don't use Safe Mode for normal operation. Many OS X functions are disabled.)
You can use the ATI Radeon HD 5770 in your system (despite what Apple says) if you are running Snow Leopard or Lion. (Your Mac Pro won't run Mountain Lion.)