The best solution for you IMO is a Radeon R9 290x that has been flashed for use in a Mac (with an unflashed one, you can still boot, but no boot screen or verbose mode). The GTX 980 Ti is a much faster card, but I found the constant reinstalling of drivers to be too much of a pain for my tastes. The R9 290x is still plenty fast; I used it for about a year until just the other day when I installed a Radeon Vega Frontier to replace, and I found both OpenCl and Metal on the R9 290x to be more than adequate for video work. I would also suggest getting a card that is sealed and blows the hot air out the back. When you replace your Apple card with a 3rd party card like this, you lose you temp sensor in the PCI, and this will help with thermal issues. Also, there's an app called TG Pro that will help you control fans manually.
My experience with Metal in Adobe software at least is that it is very fast, but it would make my card's fans blow like crazy, so I avoided it and stuck with OpenCL
FWIW, installing the Vega:
Installing a Radeon Vega Frontier Edition in a 2010 Mac Pro