There is no support for the graphic card with older versions - that is one of the limitations of Apple's drivers, and lack thereof on DVDs.
Just as the 8800GT req'd 10.5.2. The 5770 requires 10.6.4 (10.6.5+ preferably, though later 2009/2010s can boot with 10.6.3 (one or two mentioned that) the 2010s had a special build of 10.6.4.
Windows, you don't have that limitation, there is basic video output w/o or in Safe mode.
Most of the requirements on product page (ie, 2009 or later, or PCIe 2.0) almost have to read between the lines or take with a grain of salt. Except minimal builds. Just as new Macs can't boot from older OS version.
You probably while trying to work through this out to try with just one graphic card or the other, and at least you have the 8800GT so you can boot from 10.6.0 or 10.6.3 DVDs. But you can't run or boot Leopard.
You might want to setup some clean Apple only emergency boot volumes - full updated - for maintenance purposes, and to build a solid OS that you can use to clone and build a full working OS.
ATI HD 5770 MC742ZM A
One concern we had was the existence of the two Mini DisplayPorts in addition to one DVI port. We can confirm that the 2008 and 2009 were able to drive our 23" Cinema displays with the Mini DisplayPorts (using Apple's Mini DisplayPort to DVI adapter). I'm confident that the 24" LED Cinema display and soon-to-be-shipping 27" LED Cinema display will work fine as well. As for our 30" Cinema display, we used the DVI port while testing the 3G game listed above.
http://barefeats.com/wst10g2.html