My point is that you can mix and match nvidea and ATI cards in a mac pro but with mixed performance results; based on my findings below.
The 5870 is a pc card, not flashed at this time. The 2600 and 8800 are both Apple branded, the 5870 is a Sapphire card.
the following combos are tested functional in a 2008 mac pro with Lion 10.7.5:
2600 & 8800 (stock config for this box)
PC 5870 & 8800 (performance suffers in the limited testing of openGL)
PC 5870 & 2600 (expected performance for the 5870)
PC 5870 (expected performance for the 5870)
The PC 5870 drives 2 monitors even tho there are 4 connectors, per the Sapphire site this is to be expected with a passive DP to DVI adapter. Using both DVI outputs work but I've not done extensive testing to determine if there any artifacts showing based on load or application mix. I tend to run the center screen with a single card and the two side screens with the second (lower powered) card.
In all configs the system boots and everything works. Stock Mac OS, no patches, no injections, no kext mods. The PC based 5870 shows as a generic 5000 in system summary but as an HD5870 everywhere else.
If the PC 5870 is paired with an Nvidea card then openGL performance suffers but it does work.
If the PC 5870 is used by itself then it works fine but no preboot screens. Adding the 2600 allows the boot screen to be seen on one of my monitors, the same as if I used the 8800 but without the performance hit on the PC 5870.
Based on google searches the performance of the PC 5870 is similar to the 560Ti but the card can typically be had on ebay for a little less that the nvidiea card. Another alternative would be the 660TI but it's cost is currently higher.
Any of these cards would be supported as the chipsets are all in use in current Apple branded products.
At any rate this is simply to bring this thread up to date and is not intended to be a comprehensive test cycle. I followed this line of testing simply as I've installed boot camp so I can play a single game, Assassins Creed 3.
MacPro 2008 3,1 2.8GHz dual quad xenons
20GB RAM
180GB Intel 520 SSD
320GB stock Apple drive
3x1TB Samsung drives (low power 5900RPM)
1x1.5TB Samsung drive
4x2TB Samsung drives in an external eSATA RAID 5 (low power 5900RPM)
GPU's per above
Windows Experiance is 7.7 or 7.8 for everything except the drives where it is 5.9 (win7 is installed on a partition on one of the 1TB drives). So a 5 year old Mac Pro performs as well as a new gamer PC.