I might be able to add some helpful info:
My System:
Mac Pro 3,1 (2008) / 2.8 GHz 8-core, Mac OS X (10.7), 24 GB RAM / ATI Radeon 4870 + 5770
My Displays:
Apple Cinema Display: 30-inch — paired w/ ATI Radeon HD 5770
Apple Cinema Display: 23-inch (x2) — paired w/ ATI Radeon HD 4870 (flashed w/ Apple firmware)
PCI - slot 1: 5770
PCI - slot 2: 4870
PCI - slot 3: ununsed / covered by 4870 (b/c it's thick)
PCI - slot 4: eSATA controller
Unlike alexofindy, I've been able to succesfully use an eSATA controller in PCI slot 4. I have two docks—each with a 3.5" hard drive—connected to the eSATA card, and that works just fine. Based on that, I think his conclusion that "there isn't quite enough power available on the PCIe bus" for the 5770 is incorrect. I'm not sure why he's having problems, but one obvious suspect would seem to be his other video card (ATI HD 2600), since that seems to be the main difference between his system and mine.
I know nothing about that card, but I should mention my power connections for my cards (5770 / 4870) in case that's helpful to anyone. The 5770 requires only one power connection, but the 4870 requires two. There are only two (easily accessible) power connections on the logic board, so I had to purchase a Y-cable (splitter) in order to turn those two connections into three. Based on a tip I read online, instead of connecting both ends of the Y-split cable to the 4870, I connected one to the 5770 and one to the 4870. I used the standard (non-split) cable for the second connection on the 4870. This may or may not be relevant—I've never tried it the other way—but I can say that it works. (The theory, I suppose, is that the Y-split cable may not provide twice the power, so rather than channeling both potentially-deficient power sources to the same card, this arrangment dilutes the potential weakness by dividing it between both cards.)
alexofindy said, "I would guess that's why the 5770 card isn't officially supported by Apple." I don't know, but I've heard another theory that makes more sense to me—which is that Apple doesn't officially support it because it wasn't in existence when the Mac Pro 3,1 was introduced. Thus, if a user were to wipe and reinstall the system using the factory installation discs, he would have no way of getting a video signal to a display running off the 5770 without first performing some software updates (which would be hard to accomplish without a display!)
Anyway... I hope this helps provide some valuable info.
-Jason