Previous 1 2 Next 15 Replies Latest reply: Feb 12, 2014 2:32 AM by Dan Thomas (DAGWare)
Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

This post describes the lessons that I, a newbie, have learned while trying to come to grips with Compressor 4, version 4.06. I had trouble deciding whether to post this or not, because all of this information is available, in separate parts, throughout this forum. But I figured if I struggled to learn this stuff, so will other people, and perhaps my lessons can help.

 

So here's what I've learned so far. I'm a newbie, so feel free to correct any wrong assumptions or information. I'm not sure what's the best order in which to present this, so I'll just wing it.

 

I'm running Compressor 4.06, on a brand new 21" iMac. OS/X 10.8.2. I also have Final Cut Pro X.

 

All of this information is regarding using Compressor on a single machine. If you're trying to set up a multi-device cluster, take that into account as you read this post.

 

Getting Compressor to use multiple processors on a single machine

In order to get Compressor to make full use of the cool multi-core processor you have in your nice, shiny Mac, you have to use something called "Apple Qmaster", or more simply "Qmaster". The "Q" is short for "Queue", as in a queue of tasks to be executed. Here's how to get it working, and how to work around its "quirks".

 

  1. While running Compressor, click the menu item "Apple Qmaster" and then "Share This Computer". Don't let the name confuse you - we're not really going to share our computer, although we could if we wanted to.
  2. In the "Apple Qmaster Sharing" dialog:
    1. Click "Share this computer" and "as QuickCluster with services".
    2. I only have "Compressor" checked - someone else can chimed in about the "Rendering" checkbox.
    3. Click the "Options" button next to the "Compressor" checkbox, and select the number of instances you want Qmaster to use. The list shows how many cores or processors are available. How many should you choose? I have no idea, but since when I'm using Compressor I'm not doing anything else on my computer, I select all but one of them. YMMV.
    4. I have "Require these services..." unchecked.
    5. Give the cluster a name.
    6. I have "Include unmanaged services..." unchecked.
    7. If you check "Require password", you will have to enter a password before you can queue any jobs, although it can remember the password in your keychain.
    8. I don't believe I changed anything in the "Advanced" tab.
    9. Click OK.
  3. Back in Compressor, when you submit a job and the Submit dialog comes up, you can change the "Cluster" dropdown to your new QuickCluster.
  4. While a job is running, you can either view its progress in Compressor's History window, or use the button that says "Share Monitor" (which, by the way, does *not* mean you're going to share your monitor - it means your going to view the "Share Monitor" application).

 

If you want to verify that you are, indeed, using multiple cores, follow these steps:

 

  1. Run the application "Activity Monitor".
  2. Use the menu item "Window->CPU Usage", or view the bottom of the "CPU" tab in the main window.

 

Changing QuickCluster options, or turning your QuickCluster off

In order to modify or turn off your QuickCluster, you might think you could use the "Apple Qmaster->Administer Clusters" menu item. You would be wrong. If you do this, it will bring up the"Apple Qmaster" admin dialog, but you won't be able to change anything. It will insist you enter a password, when you haven't defined one. Save yourself a lot of grief, and stay away from that dialog. Trust me. On the off chance that you try it and actually get it to work without passwords, all I can say is "good for you", but don't assume it will work for others.

 

Instead, use the menu item "Apple Qmaster->Share this computer", and turn off sharing and click OK.

 

If all you wanted to do was to turn off your QuickCluster, you're done. If you only wanted to modify something, read on:

 

At this point you *might* need to close Compressor and restart it - I'm not sure, but it wouldn't hurt to do it just in case. Then create your QuickCuster again following the original steps, changing any options as desired.

 

In closing...

As always, YMMV. Feel free to tell me where I'm wrong. The only thing I ask is that you make sure you're using the same Compressor version I am (4.06), and that you actually try your steps before posting them. I don't doubt that some things work differently depending on other unforeseen configuration differences. I can only tell you what worked for me, after *many* frustrating hours of trial, error, and research.

 

Special thanks to Russ H, because most of this information was derived from various posts of his.

 

Dan Thomas


iMac (21.5-inch, Late 2012), OS X Mountain Lion (10.8.2)
  • Alchroma Level 6 Level 6 (17,765 points)

    Thanks for the info Dan.

    I would like to add that although switching stuff on and off will allow all cores to be blazing, it's a good idea to use a stopwatch and time the various configuations to get a job done.

     

    I find that my MBPro i7 late 2011 gets a job done faster without enabling Quick Clusters, so it's left on the "right out of the box " settings.

     

    As you say, YMMV.

     

    Al

  • Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

    Interesting! Silly me - I thought "more is always better". I will indeed start timing some of my processes.

     

    It will truly suck if, after spending this much time and energy trying to solve this, it turns out to not have helped. Then again, I *hate* unsolved mysteries.

     

    Thanks for the information. I think.

     

    Dan

  • Alchroma Level 6 Level 6 (17,765 points)

    I've done my share af testing for the sweet spot and I'm back to where I started.

     

    Al

  • Russ H Level 6 Level 6 (17,750 points)

    Al,

     

    My 2¢…

     

    I'm not entirely sure where the sweet spot is for me, but for encoding projects that are longer than say, 10 minutes I find there is some time savings – perhaps a third less for a 15 minute project. Because Qmaster can take several minutes to reassemble the segments, it's quite possible to lose any speed advantage for short projects. In my case, I draw that line at 5 minutes and just use "This Computer".)

     

    As Dan warned, YMMV.

     

    A lot depends, of course on your system specs. I'm running a long-in-the-tooth i5 iMac, so I only have it set to 3 instances. If you have a boat load of cores and/or hyperthreading – and these days I don't think many of us are constrained by RAM – (Apple's guidance is to keep number of instances under half the total RAM).

     

    I think testing these old truisms is great to se whether the facts line up with theory. Speaking of which, I recall David Brewer reporting some tests on Pro Res. Perhaps he'll weigh in and remind us what he found.

     

    Russ

  • Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

    Update:

     

    It appears that if you want to change options, you dont need to turn off QuickCluster then turn it back on again. All you need to do is go to the menu item "Apple Qmaster -> Share This Computer" and change the settings.

     

    Dan

  • Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

    My original post was based on my experiences with using Compressor for converting source files, not rendering output. Now that I'm experimenting with rendering output, I have a question:

     

    When I send a project from FCPX to C4, the only option I get is "This Computer", regardless of whether I have QuickCluster enabled to do Rendering or not. Any idea why this is happening?

     

    When I render this way, it looks like it's using all my cores (which is 4 on this new iMac 21"), which is fine if all I want is speed. But there appears to be no way to restrict the number of cores.

     

    Any thoughts?

     

    Dan

  • jmcm Level 1 Level 1 (10 points)

    Thank you for posting this. I was having no luck trying to setup Compressor by myself. Your post made it easy.

  • Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

    Thanks for telling me! You never know, until someone tells you. Brought a big smile to my face.

  • Jeff Timpe Level 1 Level 1 (10 points)

    Hi Dan,

     

    One more to add to the "this helped me" count. Thank you for posting a nice walkthrough, very helpful buddy!

     

    I also wanted to add that it is important to find the core sweetspot. I have a 6 core i7, which means I have 12 cores available to help render. The last thing I would do is enable 12 instances, as this will choke all the cpus and force compressor make more unessissary chunks. I find I get the best speeds from only using 5 instances. Right now I'm rendering an hour long project with 4 instances, which uses up 72% of the total cpu power according to Activity monitor. Each compresssord is using an average of 220% cpu, which is great as I would still like to keep working while it renders. However, I agree with Russ, probably only use for longer projects like anything over 5 minutes, but I need to do more tests (just got it running).

     

    Also, keep in mind where you set your storage location. It defaults to the user folder, but can be switched to another location: Apple Qmaster>Share this Computer>Advanced tab. You may want to change this location to your RAID or SSD, just make sure you keep enough room available.

     

    Thanks again!

  • Eric Weiner Level 1 Level 1 (20 points)

    Cool walkthough.  It seems like in 4.1 things have gotten diffrent though, just starting to look into it, but are quick clusters still in option at all in 4.1?  In perferances > Advanced Tap there is a additonal compressor instanes option, but when I click it I can only choose 1.  I have a 4 core imac though, with compressor 3.0 (I'm just now switching to compressor 4) I could chose up to 8 instances.

  • Dan Thomas (DAGWare) Level 1 Level 1 (105 points)

    I don't think they're available anymore. I really don't understand how it works now. There's some other more recent discussions on the subject, so I recommend you find them.

     

    Not that they answered my questions... but you might be lucky.

     

    Dan

  • Eric Weiner Level 1 Level 1 (20 points)

    Yeah I'm reading though the thread now where I'm learning everything is broken. 

     

    Apple just released an update like 20 minuites ago.  Weird because I just purchased 4.1 today.  maybe it will fix some of the problems!  (But probably not...)

  • Warwick Teale Level 3 Level 3 (580 points)

    Hi Dan, yes great post. FWIW in pre Compressor.app V4., we found the most stability for Qmaster was the MANAGED CLUSTER across distributed services nodes.. (set up with the now defunct (Dec 2013) Apple Qadminister.app. (Quickcluster according to the old  doc was an attempt to utilise the cores or instances on a single host and although it was possible to implement a quick cluster sometimes on 2 or more service nodes it seemed to not work.. quite unsatisfactory). This is why we used a MANAGED cluster. If you get some time, try it whilst you are still on V4.0 compressor. Make sure you TiCK managed cluster in the setup and use Apple Qadministrator.app

     

    Dan Thomas (DAGWare) wrote:

     

    When I send a project from FCPX to C4, the only option I get is "This Computer", regardless of whether I have QuickCluster enabled to do Rendering or not. Any idea why this is happening?

     

    When I render this way, it looks like it's using all my cores (which is 4 on this new iMac 21"), which is fine if all I want is speed. But there appears to be no way to restrict the number of cores.

     

    yep. this is old news to most in a sense. The ability to use a CLUSTER for a ROUND TRIP, or a .fcpproject has never truly been implemented or implemented to a stable state IMO.

     

    For FCP <= V7 is apparently required a licenced version of FCP 7 on each service node. Not having a spare we could use on the same subnet we could not try. Perhaps the same thing for Apple Motion.app (don't think Apple color.app was affected)..

     

    Currently, this "restriction" whether specifically implemented for licencing reasons or more likely for EASY use STABILITY, Compressor will only utilise the core on the current host "ye old This Computer / no group/cluster) to process.

     

    PROBABLE REASON: I think this might be a wise choice by Apple because in order to "RENDER" and TRANCODE an FCPX or MOTION.app project, ALL the resources MUST be available on the service nodes... a basic requirement unless COPYING into a transient area is support (FCPX, Motion seem to conditionally do this for send to Compressor.app only on the same system).

     

    Under the bonnet, the resource management system seems to be drive more by xml and soft file path usage. Thus having an easy way to have everything from Mum's itunes library, Dad's iphoto library, sister's AE project, my cousins Aperturelibrary, the odd LPX project, images and countless elements from everywhere used in the FCPX project (especially now with non-managed libraries) (w/aliases) all shared to everything begs a nightmare in the making without a true single shared arbitrated file system based over a SAN where everything resides and is accessible by all Compressor/Qmaster cluster service nodes..

     

    For most many people on this forum who have difficulty using distributed transcoding, this has been the issue why their jobs don't process on distributed Compressor.Qmaster clusters.  fwiw.

     

    Lastly, we gave up using Compressor.app and Qmaster for SHAKE.app rendering as it was very unreliable. Instead for other RENDER ONLY apps there's great render manager applications such as Pipeline (braverabbit.de) etc.

     

    Warwick

    Hong Kong

     

     

     

     

Previous 1 2 Next