As you have figured out, Compressor does not utilize the video card. It is a CPU/RAM operation.
Where a higher performance video card comes into play is with Motion and some Final Cut effects
If you run AfterEffects, Premiere or any 3d modeling software, you might want to look into the nVidia Quadro 4000 with the CUDA engine. It helps massively.
Hmm.. not to sure if thats completely true. Maybe for NOT utilised transcoding. Transcoding is not a friend of vector processors... (well at time of writing)... However....
As an example you can add a Motion project (vectors maybe) to Compressor.app and using job segmenting it will RENDER the project out say as PRORES 4444 in parallel. Very useful for loads of sets to render!
To be clear this is using MULTIPLE INSTANCEs of COMPRESSOR (not rendering).
I have the ATI 5870 in this MAC PRO and use it a lot with MOTION projects and COMPRESSOR via QUICKCLUSTER... works ok.
This was a nice tip from Mark Spencer many years ago.