Yes this post is over a year old but it did come up in my search so I thought I'd add this little cavet for anyone else who sees this.
I have run a Sapphire HD5870 Vapor-X and an Apple branded 8800GT in my 2008 mac pro (Lion 10.7.5) and they both work but the benchmarks for open GL are in the tank.
If I pull the 8800 out and put in an Apple HD 2600 XT then all is fine. If I pull the 8800 out then all is fine with the benchmarks. But putting both in drops LUXmark medium GPU only from 790 to _5_ on the 5870.
So putting both in (Nvidea and ATI) will run but somewhere the drivers get confused.
both cards, and esp the low end 2600XT are not performers, the 2600 is trouble and terrible and better off with something that supports Lion and use t he 5770.
your use of a PC card can be part of the problem I suppose to. Flashed I assume.
A GTX 570 or above if you run 10.8.2 would be a nice CUDA workhorse.
My point is that you can mix and match nvidea and ATI cards in a mac pro but with mixed performance results; based on my findings below.
The 5870 is a pc card, not flashed at this time. The 2600 and 8800 are both Apple branded, the 5870 is a Sapphire card.
the following combos are tested functional in a 2008 mac pro with Lion 10.7.5:
2600 & 8800 (stock config for this box)
PC 5870 & 8800 (performance suffers in the limited testing of openGL)
PC 5870 & 2600 (expected performance for the 5870)
PC 5870 (expected performance for the 5870)
The PC 5870 drives 2 monitors even tho there are 4 connectors, per the Sapphire site this is to be expected with a passive DP to DVI adapter. Using both DVI outputs work but I've not done extensive testing to determine if there any artifacts showing based on load or application mix. I tend to run the center screen with a single card and the two side screens with the second (lower powered) card.
In all configs the system boots and everything works. Stock Mac OS, no patches, no injections, no kext mods. The PC based 5870 shows as a generic 5000 in system summary but as an HD5870 everywhere else.
If the PC 5870 is paired with an Nvidea card then openGL performance suffers but it does work.
If the PC 5870 is used by itself then it works fine but no preboot screens. Adding the 2600 allows the boot screen to be seen on one of my monitors, the same as if I used the 8800 but without the performance hit on the PC 5870.
Based on google searches the performance of the PC 5870 is similar to the 560Ti but the card can typically be had on ebay for a little less that the nvidiea card. Another alternative would be the 660TI but it's cost is currently higher.
Any of these cards would be supported as the chipsets are all in use in current Apple branded products.
At any rate this is simply to bring this thread up to date and is not intended to be a comprehensive test cycle. I followed this line of testing simply as I've installed boot camp so I can play a single game, Assassins Creed 3.
MacPro 2008 3,1 2.8GHz dual quad xenons
180GB Intel 520 SSD
320GB stock Apple drive
3x1TB Samsung drives (low power 5900RPM)
1x1.5TB Samsung drive
4x2TB Samsung drives in an external eSATA RAID 5 (low power 5900RPM)
GPU's per above
Windows Experiance is 7.7 or 7.8 for everything except the drives where it is 5.9 (win7 is installed on a partition on one of the 1TB drives). So a 5 year old Mac Pro performs as well as a new gamer PC.
I tried this with an ATI X1900 and NVidia Quardo 4000. I have a dual Xeon (12-cpu) Summer 2010 Mac Pro with 32GB memory.
I wanted to debug CUDA, so plugged my monitor into the ATI card. OS X 10.8.3 did not see my NVidia card at all in this configuration. It was not listed under video cards nor under PCI cards in the About My Mac. So, of course, the CUDA drivers did not load when I tried to debug. When I tried to run my app, it said that no cuda devices were found.
And there WERE many negative effects in this configuration. Both iPhoto and DxO Optics Pro displayed a blank screen in the main frame instead of the photo being edited. The flash plugin in Firefox hung at 100% CPU and displayed nothing.
Total Spaces flashed any space with content violently.
Things did not work at all well.
I took it out of my 2007 Mac Pro, which died. It is really hard to find out from apple what works. The store has no video cards for sale that I can find. I have a newer ATI HD5870, but I was told that it draws too much power to use it with a Quadro 4000. I hope I can use a Quadro FX 4800 along with the Quadro 4000.
Well you could help me and us by telling us those things.
There are of course as you found out by now just two 6-pin PCIe aux power cable connections. So you need to go to external or add'l PSU to provide more like a 450W PSU to fit in the optical bay (still pain to route cables).
One of those pet peeve issues with the Mac, two 8-pin would be the same as 4 x 6-pin.
4GB VRAM works better now with ML. Use to be a 2GB limit that afer years is finally behind for most part and 3-4-6GB VRAM cards now. Two Q4Ks were an issue but doable. A 5770 + Q4K works, but a GTX 680 with two 6-pin may be more powerful than those Quadro cards. There is an over the top $2K+ Quadro 5000 on the horizon already.
If you want to number crunch CUDA, you MUST have 2 video cards to run the debugger, because you cannot drive a display and debug at the same time--it hangs the Mac dead as I have found out. So I was sad to dump my expensive ATI HD5870 to get the Quadro 4000. I was really hoping that my old lower power X1900 would work, but it does not, so I forked out for the Quadro FX4800. $1100 for education. Oh well.
The new GTX 680s are not good with CUDA and CS6. The 700s should be. Even the new ATI Mac Edition Saffaire 7950(?) is a better candidate and priced under $400. Which is why the GTX 570 was popular even after the 600s came out but are now in short supply (570/580 series).
Too old for more courses! But my "course" is to check DLloyd blog daily for updates
He does have a lot of tips and help on what to do, from 48-96GB of RAM, he does have 12-core 3.33GHz but that is overkill. A pair of 6G PCIe SSD controller cards foir large fast scratch is still a must for large files - and 4x4TB internal storage array. Don't know what he has for GPUs but is not a fan of Quadro either I am 98% sure.