Skip navigation

CLI framerate and serial jobs on a cluster

340 Views 5 Replies Latest reply: Jan 28, 2013 2:01 PM by achourasia RSS
achourasia Calculating status...
Currently Being Moderated
Jan 23, 2013 6:17 PM

Hello,

 

I have couple of questions for compressor (ver 4 )

 

1) Is there a way to specify framerate for image sequence using command line interface.

The compressor help lists the following but does not provide a way how such an option work. I tried few different ways in vain

 

-jobpath  -- url to source file.                        -- In case of Image Sequence, URL should be a file URL pointing to directory with image sequence.

-- Additional parameters may be specified to set frameRate (e.g. frameRate=29.97) and audio file (e.g. audio=/usr/me/myaudiofile.mov).

 

 

2) I have a managed cluster with 8 nodes each running 4 instances. For some reason compressor only uses 6 instances spread across many nodes and not all 32.

 

 

3) Is there a way to specify and process just one job for a given cluster? This is equivalent of serialize option but it does not seem to be available.

Currently when I submit multiple jobs few of them run at once and create a havok on NFS mounts and fail. I can limit the job queue to one but that is not ideal.

 

I would appreciate any pointers.

 

Thanks

--Amit

Compressor, OS X Mountain Lion (10.8.2)
  • Warwick Teale Level 3 Level 3 (555 points)
    Currently Being Moderated
    Jan 24, 2013 2:47 AM (in response to achourasia)

    Hi Amit, just saw your post. I assume you are passing the settings via the "-settingspath" <my_specific_job_settings_in_a_file_path_somwhere.settings" on the compressor command?

     

    If so, it's a very simple matter to specify the frame rate etc etc on the VIDEO setting itself, save it and use it.

     

    I don't recall that such "atomic settings" we're actually available in the v3 of compressor.app. I'll check later for v4. I'd be surprised  if they are. :)

     

    What I've done in the past is to simply make my own personalised settings (and destinations) up using  the Compressor.app V4 UI (save as ... my name .. i.e.  prores_noaudio_PAL" and path these file paths on the -settingspath" parm on the compressor cli. In your case I'd imagine a simple VIDEO setting for your frame rate and you're set!

     

    Compressor.app v4 makes any of your own R.Y.O settings on your ~/library/application support/compressor/settings folder. You can copy or move or link these anywhere you like and pass these customised Parms to compressor cli as above.

     

    I doubt also if these atomic settings are available as AppleScript properties, there's likely no vars like that there me thinks. . I recall the same ones exist as they do for the cli and now in compressor.app 4 they probably support Qmaster .. Yeah.. Compressor.app is apple scriptable.. It's not in the default list library so just add it...

     

    Lastly as a guess these compressor ".setting" files are probably XML base.. So you might consider to tinker with one with an editor.

     

    Anyway.. Try then"-settingspath" operand and see how u go.

     

    2) the way Qmaster schedules across unmanaged transcode nodes is likely based on how busy each node is. You should be able to see if there is a pattern simply by looking on the Qmaster job controller.log .. See share monitor.app info or use console.app look on

    your ~/library/application support/apple Qmaster/logs directory. There will be something in there.

     

    Also have a look for any errors on the cluster services logs in case the services you expects to be there are actually not.

     

    Are you using a managed cluster? Personally I have found this quite stable. Make sure u insure that those services are for managed climates only..

     

    3) yes you can specify a specific cluster on the using the "-clusterid" operand. Should you have more than one managed cluster in the farm, this is a cool way to go. Also considering the "-priority" operand usage as well for hot jobs. Make sure all submissions are low priority... It's batch.. Works great!!

     

    4) NDS mounts.. Well the simple rule is to keep them mounted on their own subnet, maker sure all destinations and sources are available to all hosts, set compressor options to copy to cluster only when ya must, and make sure the service time for NDS io READ Request is as fast as u can make it. ( jumbo frames, dedicated NICs and sinners and optimum fast read file systems... Keep other traffic away for it.. Works a treat!

     

    Should you turn something up pleased at it here for others to see. I'm certainly interested.

     

    Sorry for typos.. Just on MTR with iPhone on way home. :)

     

    Hth

     

    Warwick

    Hong Kong

  • Warwick Teale Level 3 Level 3 (555 points)
    Currently Being Moderated
    Jan 24, 2013 2:57 AM (in response to achourasia)

    Hi Amit, yeah I just had a dig around. Regarding the compressor.app personalised settings, it is XML based. The frame rate operate u might wanna chase  for frame rate is unsurprisingly "frame-rate"

     

    So say a frame rate of 25fps would be specified in the video conversion section  as "<frame-rate>R25/1</frame-rate>"

     

    Cheers mate.

    Warwick

  • Warwick Teale Level 3 Level 3 (555 points)
    Currently Being Moderated
    Jan 25, 2013 3:35 AM (in response to achourasia)

    Hi Amit, thanks for your reply. Yeah I appreciate what you are attempting to go. I recant that I'm unaware of an obvious method to dynamically pass any "framerate" parm value to the cli for compressor.app

     

    I would however state that it is VERY plausible to dynamically construct strings, values, parm files and any goodies programmatically. This as usual is a decision of your choosing. For me if I need to do a particular workflow many times, I will decide if I must automate it somehow and the subsequent trade off of supporting it , developing enhancements and the rigourous testing if it is to be used by others

     

    Your call :)

     

    In this case, I'd opt for simplicity of APPLESCRIPT or any UI or service on osx , any text manipulation, finder, compressor and file object manufacture... Speed is not a requirement as long as your workflow is automated I imagine

     

    I'd do this over messing with PERL or even UNIX bash/shell scripting because you can at least AppleScript compressor.app and for that matter the additional Qmaster stuff

     

    Now on this order:

    5) yes there sure are many XML creation apps and even within osx. I'm assuming here you'd want to build the compressor setting dynamically.  Sure you can

     

    4) NFS mounts I mentioned this because others may have a setup consideration and this is a weak link in the service transcode time when distributed.

     

    2) in regards to the algorithm and conditions deemed by Qmaster to schedule a batch over the available node services, I am not entirely convinced that its too sensitive to the actual work on a particular host, merely what it thinks it is running or has scheduled on those service nodes. I doubt it considers any load balancing other that trying to equalise work over all nodes in a cluster and their services

     

    As you are using managed cluster , why not try the TICK BOX  in the Qmaster administrator for that cluster to utilise also any "unmanaged services" . Them test a node with some services that are unmanaged and try from there? I'd be curious to know.

     

    3) targets = 1 doesn't work eh? I'll try this over the weekend.  Good idea. Of not you could again programmatically build a simple batch scheduler.  Yep now we are back to coding again :(

     

    I'll get back to I over weekend

     

    Hrh

    Warwick

Actions

More Like This

  • Retrieving data ...

Bookmarked By (0)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.