I had basically the same problem with trying to run the Mac 5770 with a Mac/PC 3870 in my 2008 Octo 3.1 MP. I was hoping I could use the component output on the 3870 (using the 9 pin ATI adapter that has component, s-video and composite outputs) for my Sony CRT Pro monitor and using the DVI to run a large screen TV for Netflicks, etc, or another smaller digital monitor. Plus, I was keeping my fingers crossed that Apple's "Color" application wouldn't see the 3850 if I had the AV switcher turning the connections off when not in use while editing in FCP Studio (Color will usually not launch if it "sees" two graphics cards connected to monitors).
Apparently the 5770 firmware has a conflict with other ATI cards: 2600, 3850 and maybe even 4870. I thought they would have a better chance of working together being ATI, but apparently not. Another user posted that he was able to get a NVIDIA GT 120 working fine with the ATI 5770 after having the same issues with the ATI 2600, but whether Color will launch is another question. I suppose I can try some older cheaper NVIDIA's maybe even some with analog outputs that the 3870 had if I could find one? Any suggestions or success stories in running a second card with the 5770?
Finally, I removed the 3870 to see how the 5770 would work by itself: the screen start-up time went from several minutes to seconds. Both monitors are very snappy and working fine with one 23" CD in the DVI port and the other23" CD using the mini-display port adapter (just the single $29 one, not dual DVI $99 type).
I don't play games or do much 3D, so I hope it was worth the upgrade price, as the 3870 had a passive cooler, was rock solid in every aspect and ran Core apps fast, but it didn't have OpenCL, so I'm hoping to see some improvement across the board in my graphics, especially the Adobe CS5 Suite.