What resolution are all the displays?
Short answer, it does work. Not sure why you had trouble.
Apple Mini DisplayPort adapters: Frequently asked questions (FAQ)
Guide to Apple graphic display adapters needed
Atlona (Dual Link) DVI to Mini DisplayPort Converter for Apple 27-inch LED and 27-inch iMac
Mac Pro (Early 2009), Mac Pro (Mid 2010), Mac Pro (Mid 2012): Issues with three displays and multiple DVI, HDMI connections
It's not the USB data that is needed to run three displays. It's the extra POWER. If you read the Apple article, they say the only solution they endorse is their US$100 Dual-Link adapters. Check the table at the end of the article.
Readers here have found that if you have two Brand-X Powered adapters, aka active adapters, it also works fine. These do not always require another USB. If designed recently, they can pick up a power pin in the Mini DisplayPort interface.
AMD (makers of the 5770 and 5870) lists the qualified POWERED adapters that meet their most stringent requirements. Look down the page and you will find several qualified Active Mini DisplyPort to various others, such as HDMI. These adapters are available from other outlets, such as Amazon, sometimes for far less than the Apple adapters.
The other desktop monitor will plug directly into the DVI port on the card and the 40 inch tv will still connect via HDMI over the second Mini Display port.
That will work only if you use an active Mini DisplayPort to HDMI adapter.
The only (someone) have found to work (2009)
If you want audio via HDMI, you can use audio from the Mac Pro's optical output with this DVI to HDMI converter:http://www.apogeeinc.net/products/item.aspx?item=2745
Active MDP to HDMI audio and video converter:
Discussion: Using three ports.
Print out the above and send it to that rep when you get yours working perhaps?
they told her that the new OS mountain lion does not support 3 monitors on 1 card
That person needs to be sent back for re-training. That configuration works well, but requires some hidden items: ACTIVE Adapters (that contain signal Boosters) from Mini DisplayPort to Single-Link DVI (for 1920 wide or under, available as StarTech or Excell) or Dual-Link DVI (for 1920 wide and up, available direct from Apple at US$100).
As you debug, test one display at a time. Once all work individually, only then try all at once.
The item in short supply is additional POWER to run the signal Boosters, so ACTIVE (powered) adapters are required.