How to set up Qmaster with Terminal in Mavericks

Question

Level 5

5,115 points

How to set up Qmaster with Terminal in Mavericks

I have a Mac Pro 5,1, 12-core with 24 GB of RAM and am trying to set up Qmaster to fully use my CPUs as a quickcluster. My current Qmasterprefs as indicated in Terminal are listed below and I'm looking for someone who can help me with the Terminal commands needed to be entered to set it up.

Martys-Mac-Pro:~ mepace$ qmasterprefs -list

Current settings:

-allowBonjourDiscovery on

-sharingType servicesAndQuickCluster

-launchContentAgent off

-log 3 truncate on

-statusMenu on

Exception: no config file entry for service "jobcontroller:com.apple.qmaster.cluster.admin"

Compressor, OS X Mavericks (10.9)

Posted on Nov 18, 2013 4:19 PM

Reply

Answer 1

Nov 18, 2013 8:22 PM in response to Martin Pace

Hey Marty--

That last line indicates that we have a file or part of a file missing. Can you look to see if you have this file:

/Library/Application Support/Apple Qmaster/qmasterservices.plist

If so, please post its contents here.

Thx,

FT

Reply

Answer 2

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 9:19 AM in response to Fred Turner

It's there.

I think the file I'm missing is jobcontroller/com.apple.qmaster.cluster.admin.plist, it's in my Time Machine backup from when I was messing around with settings. I think that once we set up a cluster a new file should be generated.

Reply

Answer 3

Nov 19, 2013 9:25 AM in response to Martin Pace

Hmmm...I'm not so sure about that. On my test machine, I didn't do anything like that (setting up a cluster on it), and it had that file. It's only a .plist file, so I think I'd restore it from your Time Machine backups and run the "-list" option again, then post that.

FT

Reply

Answer 4

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 10:21 AM in response to Fred Turner

Here's what I get after I restore the file.

Martys-Mac-Pro:~ mepace$ qmasterprefs -list

Current settings:

-allowBonjourDiscovery on

-sharingType servicesAndQuickCluster

-launchContentAgent off

-log 3 truncate on

-statusMenu on

-cluster off autorestart off servername "Marty’s Mac Pro" maxactivetargets 40 maxactivesegments 400 storagepath "/var/spool/qmaster" privatestorage off publishedstorage on storagecleanupthreshold 7 unmanagedservices on unmanagedmulticapturethreshold 0 networkinterface allinterfaces log 3 truncate on

-service "Compressor Processing" off sharing on instances 1 autorestart off unmanaged on log 3 truncate on

-service "Rendering" off sharing on instances -1 autorestart off unmanaged on log 3 truncate on

Reply

Answer 5

Nov 19, 2013 11:30 AM in response to Martin Pace

Hey Marty--

Looks like the "sharingType" flag is set correctly. "statusMenu" is also on, so that's good...hopefully you see it in the menu bar. If not, type these 2 commands back-to-back:

qmasterprefs -statusMenu off
qmasterprefs -statusMenu on

Next, let's adjust the Compressor settings (assuming you want 12 instances, 1 per core):

qmasterprefs -service "Compressor Processing" on instances 12

The other settings for that service should be retained, even though you didn't type them all out. Finally, we turn on the cluster:

qmasterprefs -cluster on

Shortly after typing these commands, the Qmaster menu should show 12 Compressor services ready to go, as well as the QuickCluster. Give Compressor a try now, and you should see the "Marty's Mac Pro" QuickCluster available as an option after you submit. If not, or if you run into errors, post any errors and changes to "qmasterprefs -list".

HTH!

Fred

Reply

Answer 6

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 11:56 AM in response to Fred Turner

I do see 12 Compressor services. When I submit a job I see this:

And I'm still not getting full usage of my 12 cores. In Activity Monitor I see:

This is essentially identicle to what I was getting before we started. Others seem to be able to get significantly higher CPU usage and faster compression.

Here's what I ended up with in qmastersprefs:

Martys-Mac-Pro:~ mepace$ qmasterprefs -list

Current settings:

-allowBonjourDiscovery on

-sharingType servicesAndQuickCluster

-launchContentAgent off

-log 3 truncate on

-statusMenu on

-cluster on autorestart off servername "Marty’s Mac Pro" maxactivetargets 40 maxactivesegments 400 storagepath "/var/spool/qmaster" privatestorage off publishedstorage on storagecleanupthreshold 7 unmanagedservices on unmanagedmulticapturethreshold 0 networkinterface allinterfaces log 3 truncate on

-service "Compressor Processing" on sharing on instances 12 autorestart off unmanaged on log 3 truncate on

-service "Rendering" off sharing on instances -1 autorestart off unmanaged on log 3 truncate on

Reply

Answer 7

Nov 19, 2013 11:57 AM in response to Fred Turner

Tested some more, and found that I had to do a little more for QuickClustering (I had only set my test system up as a node). We'll need to set the QuickCluster name, so change the last part above to:

qmasterprefs -cluster on quickclusterservername "Martys Mac Pro Cluster"

If you don't see that cluster name in your Qmaster menu, try this too:

qmasterprefs -resetCluster

Let me know how it goes!

FT

Reply

Answer 8

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 12:05 PM in response to Fred Turner

Here's my current settings:

Martys-Mac-Pro:~ mepace$ qmasterprefs -list

Current settings:

-allowBonjourDiscovery on

-sharingType servicesAndQuickCluster

-launchContentAgent off

-log 3 truncate on

-statusMenu off

-cluster on autorestart off servername "Marty’s Mac Pro" quickclusterservername "Martys Mac Pro Cluster" maxactivetargets 40 maxactivesegments 400 storagepath "/var/spool/qmaster" privatestorage off publishedstorage on storagecleanupthreshold 7 unmanagedservices on unmanagedmulticapturethreshold 0 networkinterface allinterfaces log 3 truncate on

-service "Compressor Processing" on sharing on instances 12 autorestart off unmanaged on log 3 truncate on

-service "Rendering" off sharing on instances -1 autorestart off unmanaged on log 3 truncate on

Unfortunately I'm still seeing the same performance. I also tried a

sudo qmasterprefs -restart

and did not see any change in performance.

Reply

Answer 9

Nov 19, 2013 12:08 PM in response to Martin Pace

Do you see the Cluster now in either the Qmaster menu or when you go to submit? If not, try the command:

qmasterprefs -resetCluster

It'll take a few seconds, but you should see all 12 instances and the cluster in the Qmaster menu.

FT

Reply

Answer 10

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 12:12 PM in response to Fred Turner

No, I don't. I see:

Reply

Answer 11

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 12:28 PM in response to Fred Turner

I restarted my computer and the cluster now shows up.

Reply

Answer 12

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 12:48 PM in response to Fred Turner

Submitting jobs to the cluster still gives about the same performance.

This is what has frustrated me in the past even when the Qmaster preference pane was available, I would set everything as described and see no change in performance.

Reply

Answer 13

Nov 19, 2013 1:24 PM in response to Martin Pace

Hmmm...looks like we've made progress, but not quite yet able to utilize it! Can you describe your workflow? Are you dumping straight from FCP to Compressor, or are you saving an intermediate reference file? What type of compression are you applying and what format is the source?

FT

Reply

Answer 14

Martin Pace Author

Level 5

5,115 points

Nov 19, 2013 1:33 PM in response to Fred Turner

I typically capture video using Blackmagic's Media Express (via a Blackmagic Intensity Pro PCIe card) to hard disk. Source is using QuickTime Apple ProRes 422 HQ in an HD 720p 59.94 format.

I'll edit the video using either Quicktime Player 7 (if only minor trimming is required) or FCP 7. I save the changes to disk.

Then I drag the video file from the location in Finder to my Compressor 3.5 icon in the Dock. Compressor 3.5 is set with default presets for Apple TV (H.264 for Apple TV @ 5 Mbps). Then I submit the video for compression.

Reply

Answer 15

Nov 20, 2013 10:57 AM in response to Martin Pace

I'm at a bit of a loss at the moment-- it should be working w/ all 12 cores at this point, I think. One thing that throws me is the process name that's handling the crunching for you: compressord. And just a single process at that. On my 8-core Mac Pro running 10.6.8 (sorry, I can't test it w/ Mavericks, it's a 2006), I see 8 processes called "CompressorTranscoderX" in Activity Monitor when I use QuickCluster. When I use "This Computer" instead, I only see 1 process working called "CompressorTranscoder". Checking ps in Terminal shows that they both are running "compressord" processes, but from 2 different framework bundles.

While you are processing, try this in Terminal:

ps -ax | grep Compressor

My guess is you'll see 1 process from the CompressorTranscoder.bundle and 12 from CompressorTranscoderX.bundle. And my further guess is that the process ID of the single process will match the single process you show in Activity Monitor (233 in your latest example). Can you confirm those things for me?

Some more basics to check also:

- After the restart you mentioned, do you still have the Qmaster menu and all 12 Compressor instances showing?

- Can you post the latest from "qmasterprefs -list" (after the restart)?

- What does Batch Monitor show? Do you see an entry in the "Martys Mac Pro Cluster" tab that shows the job broken into many parts and processing concurrently?

- I tried the same preset that you are using on my 8-core Mac Pro (10.6.8), and I only got 7 total job parts when using QuickCluster on a 15-min video, despite getting 20 or so using a different H.264 LAN preset. Perhaps the preset choice is affecting you (although I can't imagine why w/ two H.264 settings it'd be different)...

Fred

Reply