I have a 2008 Model Mac Pro with the original HD 5770 Gfx Card as well as a new HD 2600 XT to connect a total of 4 screens without issues.
You are mixing your cards up. The 5770 has the 3 ports and is the newer card and is (or should be) the Apple ATI Radeon HD 5770.
Yes, you should be able to support three monitors with that setup with no problems. I assume the 2600 has two ports. So put 2 monitors on it and 2 on the 5770.
You can put 3 monitors on a 5770 that but there is one caveat. You must have two active adapters on the 5770 to use all three of its ports. I'm guessing you are using passive adapters and that is why one of your monitors is blanking out.
You need to read these:
Mac Pro (Mid 2010), Mac Pro (Early 2009): Issues with three displays and multiple DVI, HDMI connections
Mac Pro (Early 2009), Mac Pro (Mid 2010): Supported display configurations
Here's a page of some of the approved adapters (active and passive) for the 5770.
Please specify the monitor connections (DVI, mini-displayport, etc.) and associated monitor size for the monitors you want to connect to the 5770.