AMD Radeon RX Vega 64 for 5,1 MP

Mojave does not support Nvidia cards so yes switching to the AMD Vega is an option. As per Install macOS 10.14 Mojave on Mac Pro (Mid 2010) and Mac Pro (Mid 2012) - Apple Support

As some background the 'newest' official Mac compatible video cards for the MacPro5,1 were the Nvidia GTX-680 Mac Edition and the AMD Radeon HD 7950 Mac Edition. Both are long discontinued but might be found secondhand.

Whilst no newer Nvidia cards will work due to the lack of suitable drivers as per the above Apple article some newer AMD cards will. These newer cards however as standard will lack Mac firmware meaning they do not start functioning until the Mac operating system has finished loading. This means functions like FileVault, holding down the option key to select a boot drive, accessing the Recovery partition etc. do not work.

However some models could be 'flashed' ones self to have Mac firmware. This covers the 7970, the R9 280 and R9 280x. More recently a company called MacVidCards who have for several years 'flashed' Nvidia cards, they have recently announced they have now managed to flash the AMD RX 580 card. See - http://www.macvidcards.com/store/p116/Apple_EFI_AMD_RX580_4GB_and_8GB_Sapphire_Pulse_for_Mac_Pro_4%2C1%2F5%2C1.html

This whilst slower than a Vega would then have the Mac firmware to enable the above pre-boot features.

Having a flashed card is not compulsory, the lack of Mac firmware merely prevents the pre-boot options.

I don't think cumbersome is the issue, it's the architecture of the logic board. The RX 64 needs 295 W power

75W from the PCIe and if you drew from both power connectors from the logic board to one card it will add 150W (75W+75W)

your still short 70W.

I dropped a 1660ti in my cMP in July and had issues with the ports (3 DP and 1 HDMI). I had to juggle drivers for most of the day. I had no display. I could boot to windows 7 which was default and hear the login but nothing on the display with 4 different monitors. If I ran it with the stock Nvida card GT120 and anytime I tried to install the windows drivers for the 1660 with both Nvida cards in or just the 120 the installer failed. I ended up putting my stock AMD 5700 back in and using both cards and then the 1660 would not operate 3 displays without one of them unable to run the native resolution on one of them and generally looking like I had the wrong prescription (glasses and medication)

Some of these issues seemed to be related to the drivers, some seemed to be issues with the architecture of the logic board as my ports are flaky as **** even in windows. I needed a GPU that had CUDA chops but I like AMD and its friendlier with macOS.