Guys, I am really confused at your trouble. The 5770 is discussed all over the boards as working 100% in every Mac Pro from 1,1 to present day with OS X 10.6.4 or above. This link, while for the "official" configuration of 2009 and later, is commonly used as reference for configuring the correct monitor setup. You can see that it refers to a DVI monitor up to 30" in the DVI port, and a second DVI monitor from the 1st MDP using a passive adapter, or 2 additional DVI monitors using powered active adapters.
http://support.apple.com/kb/HT4279?viewlocale=en_US&locale=en_US
Questions. Do you have the auxiliarly power cable from the motherboard plugged into the card? Also, are you plugged into the bottom PCIe slot and is your PCIe slot configured to be x16 which is needed to support the 5770 (bottom slot should have been this way from factory but can be reconfigured to other settings).
I researched this upgrade for a couple of months before I purchased, and everything worked 100% as planned when I received the card. Can't figure why you are having all the issues. It was literally power off, swap cards, power on, voila!
Lots more discussion here:
https://discussions.apple.com/thread/2544029?start=0&tstart=0