can xfx r6770 work on mac pro 12-core

Give it a try and find out. Be sure you have installed Mountain Lion so you have all the ATI drivers for the 6xxx series. Note you will not see the standard startup screen on the non-Apple cards unless you use VGA instead of DVI.

DVI-I from the computer (if so equipped) contains VGA directly in the four pins and the blade at the end of the connector, plus one signal from the top row of the grid:

image coutesy Wikipedia.com

Try using the VGA adaptor if it fits. At startup all you will see is a black screen because it's not an Apple card so it lacks the proper boot ROM. But after the startup is complete it should display properly. If you use the VGA adaptor then you should see the full startup process. At least that was my experience with a non-Apple GPU on my old Mac Pro.

Not all these third-party cards will necessarily work. The Nvidia GTX-660 and 680 cards appear to work from what I've read. ATI cards are always iffy. I would Google search were I you.

Hmmm. What has this to do with Linux? Apple's video drivers are already included. They should work, but of course I have no way of knowing if there is support for your card. Booting into safe mode isn't going to accomplish anything.

Maybe you need to search Google for information using this card with OS X in a 2009 Mac Pro. I don't know anything more than I have already said. In such cases I rely on using Google to find information.

The people to ask are MacRumors forum, Cindori and others.

http://www.groths.org/

http://netkas.org/

http://forums.macrumors.com/forumdisplay.php?f=1

http://www.groths.org/forum/mac-universe/graphic-cards/

http://forum.netkas.org/index.php/board,6.0.html