There are a few simple things that I would start with. I was able to get Compressor 4 working on all my cores and even able to "oversubscribe" if you will. I was even able to do it with Compressor 3.5 installed still. Not meaning to intentially, but when going to the app store and purchasing Compressor, you dont have much of an option to move it to another drive or partition with ease.
Start by installing and restarting your computer. Not sure if this is neccasary, but it seemed to clear out a few processes that were running.
Second, go to preferences and go into apple qmaster. Stop sharing all services. You wont be using this anymore.
Launch Compressor 4. Go to apple Qmaster within Compressor 4. Click on Share this computer. Click on quickcluster. You can then click the next two options for renduring and compressor and then you can control the number of instances started by clicking on the options tab.
Once you import the video and add setting click submit. Select the name of your quickcluster and share monitor will pop up. The rest is history, just like old days with 3.5.
Hope that helps. Its a simple process it just took a few minutes to figure out. The documentation for setting up a quickcluster really sucked.
Oh, once downloading and starting up Compressor 4 for the first time, you should go get updates, looks like there was a patch for some of the video formats.
Cant promise it will fix everything but I couldnt even find a simple setup doc or thread.