How to share a job with Compressor 4.1?

Hey Steffen Bendix,

Thanks for the question. I found information relevant to your question in the Compressor User Guide. You may find the information helpful:

Compressor User Guide

manuals.info.apple.com/MANUALS/1000/MA1667/en_US/compressor_4.1_user_guide.pdf

Note: Refer to page 101

Thanks,

Matt M.

Or online here.

http://help.apple.com/compressor/mac/4.1/#cpsr4748621f

Specifically what codec are you transcoding your original to?

You have to set up file sharing on both computers, I would think. Double check that.

Hi Steffen, if you haven't resolved your issues here, check for the state of the errors in the COMPRESSOR.logs via console.app

in ~/Library/Logs/Compressor :

• contentcontroller:com.apple.qmaster.contentagent.log
• jobcontroller:com.apple.compressor.cluster.admin.log
• qmasterd.log
• requestprocessor:com.apple.compressor.contentcontroller.log
• servicecontroller:com.apple.stomp.transcoder.stomp.log
• servicecontroller:com.apple.stomp.transcoderx-1.log
• servicecontroller:com.apple.stomp.transcoderx-2.log
• servicecontroller:com.apple.stomp.transcoderx-3.log
• servicecontroller:com.apple.stomp.transcoderx.log
• unfsd.log

there might be errors in there that explain why the compnents are not there to invoke the trancode.

Post your results for others to see.

Warwick

Hong Kong

Hi Steffen, hats off to you for gathering this valuable information! I'm going to title this post:

Setup Distributed Node Processing for Distributed Segmented MULTIPASS Transcoding in Compressor.app V4.1 (2013 version)

Summary:

A quick look at those logs of yours.., Qmaster is having trouble accessing its cluster storage and probably your transcode source and target elements.

This is a bit of a giveaway - looks like the part time helpers at Apple didn’t look at it hard enough 🙂

msg="CSwampService::startupServer: servicecontroller:com.apple.stomp.transcoderx couldn't advertise server, error = error: DNSServiceRegister failed: CRendezvousPublisher::_publish"/>

<mrk tms="412112937.953" tmt="01/22/2014 20:48:57.953" pid="1975" kind="begin" what="log-session"/>

<mrk tms="412113195.964" tmt="01/22/2014 20:53:15.964" pid="1975" kind="begin" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Processing."></mrk>

<log tms="412113209.037" tmt="01/22/2014 20:53:29.037" pid="1975" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared] failed, error = 61"/>

Let’s look at this and then propose a clean method of establishing and consolidating your cluster.

Simply the Bonjour service is having a hard time trying to find you and also qmaster been running ragged trying to mount your Cluster 21D262F0-BF7EC314 storage.

Let's fix it.

Basics for the above with Compressor v4.1 and Qmaster.

• much has been abstracted from the user to help easy implentation and use. This is good me thinks! 😉
• avoid ticking every option that is available on each host , such facilities aggravate and add to the complexity of your workflow environment
• isolate the network subnets to use for all host access, file system paths, communication, diagnosis, monitoring and finally data transfer (see later)
• use source objects that will develop segments that will be distributed for processing. A 3 minute clip generally won't segment to cause distribution.
• review any workflow gains by distributed transcoding: slow node hols up process and additional time to assemble qt segments. A cluster dedicated to an iMac or macpro can often be faster. (Have several clusters defined and submit accordingly (long , most and short )!!)
• All elements/objects used in the source and any target folders SHOULD (not must) be mounted and accessible by each Qmaster node. You can use sym links I recall. For reasons of efficiently and easy of diagnosis.
  

So.. I'd propose you try and test your setup as follows .

Try this and start from beginning. Do your best to perform these work instructions. Try also not to deviate if u can

Simple Architecture Approach:

• Your main Macbookpro or main work mac (refered to by you as "master") shall be designated the qmasterd controller that services batch submissions AND that also provides transcode services.
• The other macs ("service or slave nodes) will ONLY perform transcoding services and will NOT accept batch submissions. The slaves / service nodes will not be able to send their jobs to your master controller for transcoding for example.

Keep it simple! and please follow these steps.

Step 1: Quiesce your clusters and Qmaster

1. in Compressor.app v4.1 / Preferences / Shared Computers, stop / disable all hosts (both your macs) from automatic file sharing - tick it OFF (it causes issue u have).. More later
2. In Compressor.app v4.1 / Preferences / My Computer, stop / disable all hosts (both your macs) stop allowing others to add batches to your host. Slide to OFF
3. On all hosts, quit or force out compressor.app v4.1
4. On all hosts (macs) use activity monitor.app or unix ps command to Force Quit ( or kill) any currently running qmasterd task and any compressord tasks if you can.
5. On all hosts, purge | clean out | delete the Qmaster and compressor structures. This is documented by several of us on this forum but fundamentally you want to preserve your settings and destination templates and delete the rest. Do these sub-steps on all hosts where u intend to deploy compressor/Qmaster for your distributed transcode processing

6. Check the activity monitor and confirm there are no compressord sub tasks running. Qmasterd might be running. That's ok

Step 2: set up your dedicated network for your transcoding cluster .. Important!

In this step you will isolate your cluster network to a dedicated subnet. BTW ,o DNS is needed unless you get exotic with many nodes!

You will:

• use the macs Wifi network as your primary network for NON TRANSCODING work such as email, internet , iChat, iCloud and bonjour (.local) I'm assuming u have this in place
• use the Ethernet on your macs as your dedicated Qmaster cluster subnet.

For this procedure will make an IP address range 1.1.1.x subnet and manually assign the IP addresses to each host. Ofcourse you can use a smart DHCP router if u have one or a simple switch and use OSX SERVER.app erver 10.9 ($HK190 , €19) on your MAcbookpro... The later for another time 😎

e) on each host (mac)system preferences/network to make a new network configuration (so that you can fall back incase of error) : using system preferences/network make a new network location on each mac

• - edit the Location listbox, edit and DUPLICATE the current location
• - select and over type the name and change it to "qmaster work" or some name u like save and close the dialogue
• - back in sys prefs / network select your new location "qmaster work" (from the location list box and "apply"
• - now click the gear wheel icon on lower left and Reorder the network interfaces so that the WIfi is top (first) followed by Builtin Ethernet .
• - click "apply"
• - confirm Ethernet is still green status
• Do this on each host (mac) .. The slave/service nodes

Step 3: mount all filesystems over the Ethernet 1.1.1.x subnet that are to be used for transcoding source (input | read ) and target (output | to be written )

Simplicity is important at this stage to make sure you know what being accessed. This is one reason for disabling all the automatic compressor settings

You will use the Finder's "Connect to Server" (+k) from each slave (server) node to access the source and target filesystems on your master mac for the transcoding.

These can be saved as favourites in the "Connect to Server" dialogues

Do this:

A) locate the volumes / filesystems and folders on your mac master where your source objects is contained. Do the same for where the final distribution transcode is to be written with you user access.. "Steffen"

B) On each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs

• mount the Source folder. Finder / Go / Connect to Server or +K
• enter "Steffen@1.1.1.1//Users/steffen/movies/my-fcpx-masters. ( choose you source directory path) .
• Click connect & password and use the "+" sign to save as favourite

- mount the target folder. Finder / Go / Connect to Server +k

- enter "Steffen@1.1.1.1/users/movies/my-fcpx-transcodes. ( choose your target directory path) . Click connect n password and use the "+" sign to save as favourite

Do these for all your slave macs. Remember you are specifying file paths over the 1.1.1.x subnet

Obviously make sure your slaves have read and write access. Yes and you could also mark these folders and Shared in Finder info then everyone can see them ... your choice

C) verify your access: on each slave use the finder to create a ftest older in those recently mounted volumes. Delete the test folder.

So now all your networks and workflow folders are all mounted and accessible by your slave hosts and it's user 😉

step 4: Set up Compressor v4.1 and Qmaster

Care is initially required here NOT to click needlessly on options.

Recall that you purged most of the compressor.app v4.1 state information on step 1. ?

Compressor.app v4.1 will appear new when u start it.

1. on the master mac 1.1.1.1 , launch compressor.app v4.1
2. open compressor v4.1 preferences (command+,)
3. using compressor.app V4.1 preferences:
   • My Computers tab: turn OFF . Don't allow others to process on this computer
   • Share Computers tab: UNTICK (disable) automatically share files.
   • Advanced tab: WHen Sharing My Computer listbox: select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble👿
   • Do not specify additional instances yet! Less to troubleshoot
4. On each slave mac 1.1.1.2 -1.1.1.x
   • launch compressor.app v4.1
   • open compressor v4.1 preferences (command+,)
   • using compressor.app preferences:
   • My Computers tab: turn ON (yes ON) to allow others to process their stuff on this slave computer
   • Share Computers tab: UNTICK (disable) automatically share file
   • Advanced tab: WHen Sharing My Computer listbox select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble
   • Do not specify additional instances yet! Less to troubleshoot !
5. On the master mac, 1.1.1.1
   • using Compressor.app v4.1, select destinations and add a new CUSTOM destination and navigate the dialogue to your target folder you specified in step 3b ~/movies/my-fcpx-transcodes as an example.
   • Use this custom destination on the batch later
   • in Compressor.app V4.1 preferences/Shared Computers, click the plus "+" sign in bottom left cornet to create a new cluster called "unnamed ".
   • - click in the name and change the name to "Steffenscluster" (I'm not connected to my network as I do this..)
   • - tick on the slaves 1.1.1.2 to 1.1.1.x to add the to your new cluster .. assumethee are on the right in a list.
6. Your cluster "Steffenscluster" is now active!
7. Care care! One more thing to do. You SHOULD make the cluster storage for master 1.1.1.1 available to all your slaves. This is important for this test !!
8. On the master 1.1.1.1, Use the finder to verify that you have these directories built from compressor.app v4.1
   • /Users/Shared/Library/Application Support/Compressor
   • and your own home directory: ~/Library/Application Support/Compressor
   • Dig deeper for the /storage folder in each to see the hexadecimal named folder that represents this cluster "Steffencluster"!
   • These should be manully mounted on each slave. Recall we DISABLED automatic file sharing.
9. on each slave mac 1.1.1.2 - 1.1.1.x, mount the masters cluster storage file systems. Do this for verifying access of each cluster Slave
   • on each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs
   • mount Qmaster cluster storage folder
     • Use Finder / Go / Connect to Server or +k
     • enter "steffen@1.1.1.1/Users/Shared/Library/Application Support/Compressor/Storage
     • Click connect & password and use the "+" sign to save as favourite
   • - mount users Qmaster cluster storage folder
     • Use Finder / Go / Connect to Server or command+k
     • enter "steffen@1.1.1.1/Users/steffen/Library/Application Support/Compressor/Storage

     Click connect & password and use the "+" sign to save as favourite

10. This you may have 4 new network volumes (file systems) mounted on each mac slave over your dedicated 1.1.1.x subnet!

Step5: submit a job

1. On master mac 1.1.1.1 launch the new compressor.app v4.1 "Network Encoder Monitor " .. Use +E . new in COmpressor.app V4.1 😀
   • You should see all your nodes all cluster service points for your master and slaves. Heres you see just one (this macbookair!)
   • On each host (mac) Launch the activity monitor.app and filter compressord .. There they are on each host!
2. Nearly there.
   • On the mac that's the master 1.1.1.1 (controller )
   • Submit a job:
   • Use finder to move some footage etc that more than 15 mins for example into your source directory folder from step 3b (eg ~/movies/my-fcpx-masters.)
   • In compressor.app v4.1 Add this +I to the batch
   • Drag your custom destination on to the batch
   • Set your encoding setting (apple devices best)
   • Open the inspector in compressor.app and select "video" and make sure MULTIPASS is checked. Then change to frame controls at your leisure. Better means slower
   • Select the General tab and make sure JOB SEGMENTING is ticked!
3. Now cross fingers and submit it (or +B)

Step 6: Monitoring the Workflow Progress😁

In compressor.app v4.1 Use the active tab to watch the progress

1. Open the disclosure triangle and see the segments
2. Unfortunately u can really see which node is processing. (No more share monitor .. btw for those who care.. thats buried now in /Applications/Compressor.app/Contents/PlugIns/Compressor/CompressorKit.bundle/Contents/Embedde dApps/Share Monitor.app/Contents/MacOS/Share Monitor
3. Look at the network encoder monitor (cmd + E) to see the instances processing your work

Lots of small and overdetailed steps here Steffen and its worth working through.

Simply all these things need to be availble to get your cluster to work EVERYTIME.

I might update this and post a more detailed disalogue/transcript on my blog and post it here.

Epilogue:

I for one rather like the new compressor.app V4.1. Qmaster is buried and works well when not teased or unintentionally fooled.

I would like the ability to :

• specify the location of the qmaster cluster storage rather than have it in the root systems file system - I used to have it on my disk array
• compressor to be applescriptable

Post your results for others to see.

Warwick

Hong Kong

Hi Steffen, well it's better having a deal of detail than advice that's based on the drone of "removing prefs" and "repairing disk permissions" based on some but not all "folk lore" and perhaps myths!

(Sad that the screen snapshots did not consolidate into my previous post.. my error)

To your latest effort. It certainly looks like you had some pleasure with this, although it was brief. 😠

I am certain that this latest issue is simply related to the ACCESS and the availability of the objects (resources) you need for your transcoding transactions.

Steffen Bendix wrote:

.....snip snip. It worked only once. I did every single step. As expected, the crux was the mounting of the network volumes. Mounting with "steffen@1.1.1.1/Users/steffen/Library/Application Support/Compressor/Storage" failed. I instead added the Storage folders and the source and destination folders manually in the Shared preference pane of Mac OS X and mounted the folders with "steffen@1.1.1.1/". Up came a dialog where I could select all four shared folders. There was no storage folder in /Users/shared/Library/Application Support/Compressor/ so I added it manually.

This may be true because initially qmaster has likely utilised /users/shared/application support/compressor/storage/xxxxx. It certainly does not hurt anything if you add this structure with the correct access and permissions.

Steffen Bendix wrote:

The slave Mac again gives the error message in the Network Encoder Monitor that it could not find the source file.

Yes, then there is certainly an access issue here that's likley very very easy to fix. THe detail of the resource missing will be in the host SLAVE (1.1.1.2) /users/steffen/library/logs/compressor/ check these logs out ... especially the com.apple.stomp.transcoderx. logs use the console.app .

In the log above you can see can see the successful access of the host. In you case, I'd be certain there was an error in eht slave for this.

This is merely a file path or access to the source r the target.

worth checking. Post here if you like.

When you get your system stable, THEN bugger around with instances and some automation.

I use shared dedicated directories for the client jobs I do. This way I have everything accessible from each host. I also use destination templates which is a huge workflow help.

Steffen Bendix wrote:

. I am so tired of this Compressor clustering thing.

Yeah I can imagine 😟

Stick with it though mate. It works very very well when its sorted out.

I believe Apple's intention was to make it very very easy to use. They have done a good job so far with compressor.app v4.1 (2013)

I think the available Compressor.app V4.1 documentation is very light and over simplified for resolving issues. It's compounded by urban myths and legends and some good intentioned "delete the prefs" people without the time or perhaps the need to look into the problem.

FWIW: I am asked to help out clients with this so I keep on top of it as best as I am able such as Russ and a few others.

Keep trying Steffen and I beleive you will be not only happy when you get it all working, and additionally you will make SOME of your workflow more efficient and become more knowledgeable in the process.

Post your results for others to see.

Warwick

Hong Kong

HI Steffen, yes you have seen these MOUNT failures in the compressor stomp logs.. let's have a quick look...:

Steffen Bendix wrote:

<mrk tms="412286609.127" tmt="01/24/2014 21:03:29.127" pid="2123" kind="begin" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg="Processing."></mrk>

<log tms="412286622.155" tmt="01/24/2014 21:03:42.155" pid="2123" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35/shared] failed, error = 61"/>

The first failure that you have documented here in Qmaster on your slave where it seems to be trying to mount its cluster storage for your cluster whose internal (global) name is ) 64DE4D86-4736EB35 on the MASTER that I recall should be IP address 1.1.1.1 ? (y/n)?

Instead its trying to mount your cluster storage at /users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35 / ...... that it believes is on the local host (Slave @ 128.0.0.53?).. should not be.? 😐

Recall you have set automatic file sharing to OFF in the compressor qmaster cluster setup. This helps to get this working and understand what is going on here

By the way, did you have a dedicated path set to BUILTIN ETHERnet on the master (@ 1.1.1.1) and the slave ( @ 1.1.1.2) in the compressor cluster (group) parms:? please check else it will take any path and that might work however you want to push these transactions over the 1.1.1.x subnet me thinks!

Here's an example:

Verify that you have specified "Built-in Ethernet" or its name on ALL (Both) of your machines for the subnet 1.1.1.x. 😀

Series 01 Questions and Actions to try:

1. do you have a dedicated network path from your slave (1.1.1.2) to you other mac at 1.1.1.1? (should be ok, please verify from master IP @ 1.1.1.1). (As above)...

   try this from the slave ( @ 1.1.1.2) using Terminal command: traceroute -s 1.1.1.2 1.1.1.1 (us ethe -s switch to and the 1.1.1.2 parameter to PUSH the traceroute from the slave's 1.1.1.2 interface .. your ethernet I recall).. should get one line that looks ok. IF a no-route or similar, please set up your ethernet point to point. should be a no brainer

2. from your slave machine (1.1.1.2) can you SUCCESSFULLY NETWORK mount your private cluster storage on to you slave at 1.1.1.2? (note that resides on your master mac (1.1.1.1) at paths:
   • /users/steffen/Library/Application Support/Compressor/Storage

     use Finder / go / "connect to server" or +K - enter as text /users/steffen/Library/Application Support/Compressor/Storage and "connect" - put your logonid and or password in as/if requested in the AFP dialogue. The file system should mount on your slave mac (see the desktop).

   • /users/Shared/Library/Application Support/Compressor/Storage (if it exists) else bypass the next sub-step.

     use Finder / go / "connect to server" or +K - enter as text /Users/Shared/Library/Application Support/Compressor/Storage and "Connect" - put your userid and or password in if | as requested in the afp dialogue. The file system should mount on your slave mac (see the desktop).

   • you probably have two filesystems mounted on your slave called "Share"... open them and see if you cna add and delete a file or folder.
   • recall in the previous posts I suggested you make a favourite of these in the "connect to server" dialogue.
   • FWIW - when this settles down it will be possible ot have these automatically mounted.. Rather than let it default, I just have a rudimentary Automator.app workflow .. click and forget.. then it's completed.. this later if you're interested

so far so good? - (... over..)😀

Steffen Bendix wrote:

After this I mounted the folder "64DE…" directly and go this:

<mrk tms="412289024.344" tmt="01/24/2014 21:43:44.344" pid="2123" what="service-request" req-id="A84C4AF0-7542-4ED9-A5B3-CA91901EEBCF:1" msg="Error: unrecognized request."></mrk>

<mrk tms="412289065.206" tmt="01/24/2014 21:44:25.206" pid="2123" kind="begin" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg="Processing."></mrk>

<mrk tms="412289070.244" tmt="01/24/2014 21:44:30.244" pid="2123" kind="end" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg=" error: Ausgangsdatei „/Users/steffen/Downloads/Compressor/Source/Test.mov“ wurde nicht gefunden"></mrk>

The last line in German means that the source file was not found. This is nonsense because the folder "Source" is mounted and has read/write access from the slave computer.

Thanks for the translation BTW mate!..

Series 02 Questions and Work Instructions.:

1. is the file system where your source (I assume its a source .. good name anyway!) for "Test.mov" is MOUNTED on the SLAVE system (@ 1.1.1.2) ? It's complaining about it that the qmaster prefight (setup) can't find the path mentioned above to your source(Test .mov) on your master @ 1.1.1.1
2. . To be precise and simplistic..
   • confirm for this post that the file path in the error message is actually ar Master (@ 1.1.1.1) @1.1.1.1/Users/Downloads/Source/Test.mov?

     yeah I know.. I'm just confirming with you. 😊

   • confirm that you have this file path (Folder is "Source") mounted on your slave mac @ 1.1.1.2) from your Master @ 1.1.1.1 ?
   • have a look from slave @ 1.1.1.2
     • via Terminal as: ls -1 /Volumes
     • via Terminal as df
     • in Finder under Shared sidebar and on the desktop?
3. IF not then from the slave @ 1.1.1.2, MOUNT the file path where your source is :

   Finder / Go / "Connect To Server" or +K and add the path as :

   • afp://steffen@1.1.1.1/users/steffen/Downloads/Compressor/Source
   • make a favourite in the dialogue for your convenience. 😍
4. (BTW.. when you sort this out, you can utilise some simple setup in Automator.app so that you donthave t bugger wi this evry time.. it's excellent!)

   (later , use a set up known MAster directories, and Client distribution directories and make them into TEmplates (destinations etc) and it will always work. .. later for another thread ).

IF you have tried the above, do you still get this error that you reported above?

Do you get the same error for the 1.1.1.x path? If so check for access (+i) on the cluster. free free to add an ACE (access control entry) the the ACL (access control list) and give yourself steffen read+write and access if you don't have it for some bizarre reason! 😕

In case you do:

Do this in the Master @ 1.1.1.1 where you Source resources are:

(You should NOT have to do this.. where you see my name it should already have steffen as owner and read write).

Submit your batch again.

Summary:

To make this work all around using a distributed node, it's a good idea to have ALL the source elements required and target output paths and the cluster all shared to all hosts on the cluster (you @ 1.1.1.2 slave mac) as I remarked in my earlier posts.

One would anticipate that all tihings being transparant and easy that Qmaster / compressor would attend to all tis. It does when you have evertything available. In this case we have purposely NOT used automatic file sharing... you can see why doing this MIGHT cause issues of performanace , access and diagnosis.. this is why I dont utilise it. INstead I manaual maount all my source and targe resources.

Others reading this post may critique my procedure and note it unnecessary... and this is to offer a fair opinion that QMaster should take care of all this by copying resources to the shared cluster (spool) and permitting nodes to access from there!😀. I say "Yes", to those opinions... Qmaster may do this, however it's time consuming and seems to be error prone at times, hence this manual approach used my others and myself.

Give it another go mate! 😎 You're nearly there.

Post your results back for others to see.

Warwick

Hong Kong