1 2 3 Previous Next 39 Replies Latest reply: Mar 21, 2014 7:07 AM by aidassup
Steffen Bendix Level 1 Level 1 (40 points)

Can anyone explain how to set up Compressor on two or more computers to share a coding job? I was never successful, neither with the old versions nor the new one. I have connected two computers running Mavericks via Ethernet. They appear in the preferences list of Compressor as inactive and can be selected (with a tick). Starting a job produces no error. Only the little network monitor window shows some activity: "inactive" (in white or yellow), sometimes: "not found" (in red). The computer which sends the job waits endlessly.

 

I deactivated the firewall, connected the computer with DHCP or fixed IP but no success. What else do I have to do?


iMac (27-inch, Late 2012), OS X Mountain Lion (10.8.4), 16 GB, Nvidia GeForce GTX 680MX
  • 1.
    matt00 Community Specialists Community Specialists (3,790 points)

    Hey Steffen Bendix,

     

     

    Thanks for the question. I found information relevant to your question in the Compressor User Guide. You may find the information helpful:

     

    Compressor User Guide

    manuals.info.apple.com/MANUALS/1000/MA1667/en_US/compressor_4.1_user_guide.pdf

     

    Note: Refer to page 101

     

     

    Thanks,

     

    Matt M.

  • 2. Re: How to share a job with Compressor 4.1?
    BenB Level 5 Level 5 (7,825 points)
  • 3. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    Thank you both for the replies. I already read those information. As I wrote, the shared computers are displayed in the preferences and can be selected. When a job is started, the master computer tries to distribute its footage but the connected slave doesn't do anything. It doesn't get the chunks of the material. It seems it can't get proper access to the shared volume. I tried different things: Ethernet and WIFI, manual file sharing and automatic sharing, Firewall on or off etc. Nothing helps. I never got a distributed transcoding work. Wait – no, it worked  only one time (approx. 2 years ago). The slave got the work pieces for a short time and stopped after doing a bit of transcoding. It never ever worked again since then.

     

    This is what I found in the system log:

     

    Dec 21 09:00:31 localhost kernel[0]: "vm_compressor_mode" is 4

    Dec 21 09:10:34 Friedward.local WindowServer[207]: disable_update_timeout: UI updates were forcibly disabled by application "Compressor" for over 1.00 seconds. Server has re-enabled them.

    Dec 21 09:10:34 Friedward.local WindowServer[207]: common_reenable_update: UI updates were finally reenabled by application "Compressor" after 1.12 seconds (server forcibly re-enabled them after 1.00 seconds)

    Dec 21 09:11:44 Friedward kernel[0]: nfs connect socket error 61 for 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared

    Dec 21 09:11:46 Friedward kernel[0]: nfs connect socket error 61 for 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared

    Dec 21 09:11:57 Friedward.local qmasterd[681]: CDNSharedStorageServer::subscribeNotification: CException: CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared] failed, error = 61

    Dec 21 09:12:54 Friedward.local Compressor[2308]: [CQNetworkEncodingServiceMgr getServiceLog]: exception caught: type = 33554432, error = 0, message = Error: serverURL is null!

    Dec 21 09:12:54 Friedward.local Compressor[2308]: [CQNetworkEncodingServiceMgr openAllLogs]: exception caught: INVOKING METHOD OR ACCESSING MEMBER FOR NULL REFERENCE

    Dec 21 09:14:32 Friedward com.apple.launchd.peruser.501[389] (com.apple.Compressor.SharingHelper[471]): Exited: Killed: 9

    Dec 21 14:21:28 localhost kernel[0]: "vm_compressor_mode" is 4

    Dec 21 14:50:48 Friedward com.apple.launchd.peruser.501[399] (com.apple.Compressor.SharingHelper[464]): Exited: Killed: 9

    Dec 21 14:51:41 localhost kernel[0]: "vm_compressor_mode" is 4

  • 4. Re: How to share a job with Compressor 4.1?
    BenB Level 5 Level 5 (7,825 points)

    Specifically what codec are you transcoding your original  to?

  • 5. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    It doesn't matter which codec I try to transcode. I used diverent: mpeg4, ProRes, h.264. The error message on the slave is "Shared storage mount failed: host = master computer.local, path = …" ("master computer" means the name of the master computer and "path" is the path to master computer/Users/user/Application Support/Compressor/Storage/cryptic folder name

     

    When I use a custom folder, Compressor on the slave can't access the footage. It is the same behavior on three different computers.

  • 6. Re: How to share a job with Compressor 4.1?
    BenB Level 5 Level 5 (7,825 points)

    You have to set up file sharing on both computers, I would think.  Double check that.

  • 7. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    Believe me, I double / tripple / x checked that. Of course it is enabled. I even mounted the specific storage folders on the other Mac. The only volume I can't see from the other side is that temporary Shared volume. May this be the reason? I cannot believe that three different computers have the same problem. Does it work with your computers? Can you write a detailed step-by-step instruction here, please?

  • 8. Re: How to share a job with Compressor 4.1?
    Warwick Teale Level 3 Level 3 (575 points)

    Hi Steffen, if you haven't resolved your issues here, check for the state of the errors in the COMPRESSOR.logs via console.app

    in ~/Library/Logs/Compressor :


    • contentcontroller:com.apple.qmaster.contentagent.log
    • jobcontroller:com.apple.compressor.cluster.admin.log
    • qmasterd.log
    • requestprocessor:com.apple.compressor.contentcontroller.log
    • servicecontroller:com.apple.stomp.transcoder.stomp.log
    • servicecontroller:com.apple.stomp.transcoderx-1.log
    • servicecontroller:com.apple.stomp.transcoderx-2.log
    • servicecontroller:com.apple.stomp.transcoderx-3.log
    • servicecontroller:com.apple.stomp.transcoderx.log
    • unfsd.log

     

    there might be errors in there that explain why the compnents are not there to invoke the trancode.

     

    Post your results for others to see.

     

    Warwick

    Hong Kong

  • 9. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    Apple Support wrote that they never had this issue before. There are hardly any errors in the logs. This is what I found.

     

    On the slave computer in servicecontroller:com.apple.stomp.transcoderx.log

     

    <mrk tms="412112622.278" tmt="01/22/2014 20:43:42.278" pid="905" kind="begin" what="log-session"/>

    <log tms="412112622.380" tmt="01/22/2014 20:43:42.380" pid="905" msg="CSwampService::startupServer: servicecontroller:com.apple.stomp.transcoderx couldn't advertise server, error = error: DNSServiceRegister failed: CRendezvousPublisher::_publish"/>

    <mrk tms="412112937.953" tmt="01/22/2014 20:48:57.953" pid="1975" kind="begin" what="log-session"/>

    <mrk tms="412113195.964" tmt="01/22/2014 20:53:15.964" pid="1975" kind="begin" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Processing."></mrk>

    <log tms="412113209.037" tmt="01/22/2014 20:53:29.037" pid="1975" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared] failed, error = 61"/>

    <mrk tms="412113216.015" tmt="01/22/2014 20:53:36.015" pid="1975" kind="end" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg=" error: Aktivierung von Shared-Storage fehlgeschlagen: Host = Idefix.local, Pfad = /Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared"></mrk>

    <mrk tms="412113313.994" tmt="01/22/2014 20:55:13.994" pid="1975" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412113374.000" tmt="01/22/2014 20:56:14.000" pid="1975" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412113376.963" tmt="01/22/2014 20:56:16.963" pid="1975" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412113434.007" tmt="01/22/2014 20:57:14.007" pid="1975" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Error: unrecognized request."></mrk>

     

     

    On the master computer in requestprocessor:com.apple.compressor.contentcontroller.log

     

    <log tms="412113087.534" tmt="01/22/2014 20:51:27.534" pid="5014" msg="Destination file is local - we will try hard-linking."/>

    <log tms="412113087.534" tmt="01/22/2014 20:51:27.534" pid="5014" msg="Hard-linking result file to its destination failed, will need to copy."/>

    <log tms="412113087.534" tmt="01/22/2014 20:51:27.534" pid="5014" msg="Local-copying result file to its destination."/>

    <mrk tms="412113092.550" tmt="01/22/2014 20:51:32.550" pid="5014" kind="end" what="service-request" req-id="B0A08E85-1F0B-4EA1-BA8D-63AC3B5C72AC:7" msg="Postprocessing service request end."></mrk>

    <mrk tms="412113092.576" tmt="01/22/2014 20:51:32.576" pid="5014" kind="begin" what="service-request" req-id="161573C5-0200-4BC2-913B-81466E8E9F8B:3" msg="Postprocessing job."></mrk>

    <mrk tms="412113092.629" tmt="01/22/2014 20:51:32.629" pid="5014" kind="end" what="service-request" req-id="161573C5-0200-4BC2-913B-81466E8E9F8B:3" msg="Postprocessing job request end."></mrk>

    <mrk tms="412113180.654" tmt="01/22/2014 20:53:00.654" pid="5014" kind="begin" what="service-request" req-id="9807157D-D9C4-4968-BA04-CD870E25209A:1" msg="Preprocessing job."></mrk>

    <log tms="412113180.662" tmt="01/22/2014 20:53:00.662" pid="5014" msg="Source file is local - we will try hard-linking."/>

    <log tms="412113180.685" tmt="01/22/2014 20:53:00.685" pid="5014" msg="Hard-linked source file to cluster storage."/>

    <mrk tms="412113185.659" tmt="01/22/2014 20:53:05.659" pid="5014" kind="end" what="service-request" req-id="9807157D-D9C4-4968-BA04-CD870E25209A:1" msg="Preprocessing job request end."></mrk>

    <mrk tms="412113185.691" tmt="01/22/2014 20:53:05.691" pid="5014" kind="begin" what="service-request" req-id="B7499461-6797-4477-86A2-D6CBE6E4EDA0:1" msg="Preprocessing."></mrk>

    <mrk tms="412113190.720" tmt="01/22/2014 20:53:10.720" pid="5014" kind="end" what="service-request" req-id="B7499461-6797-4477-86A2-D6CBE6E4EDA0:1" msg="Preprocessing service request end."></mrk>

     

    I can't find any errors in the other logs on both computers.

     


  • 10. Re: How to share a job with Compressor 4.1?
    Warwick Teale Level 3 Level 3 (575 points)

    Hi Steffen, hats off to you for gathering this valuable information!  I'm going to title this post:

    Setup Distributed Node Processing for Distributed Segmented MULTIPASS Transcoding in Compressor.app V4.1 (2013 version)


    Summary:

    A quick look at those logs of yours.., Qmaster is having trouble accessing its cluster storage and probably your transcode source and target elements.

     

    This is a bit of a giveaway - looks like the part time helpers at Apple didn’t look at it hard enough

     

    msg="CSwampService::startupServer: servicecontroller:com.apple.stomp.transcoderx couldn't advertise server, error = error: DNSServiceRegister failed: CRendezvousPublisher::_publish"/>

    <mrk tms="412112937.953" tmt="01/22/2014 20:48:57.953" pid="1975" kind="begin" what="log-session"/>

    <mrk tms="412113195.964" tmt="01/22/2014 20:53:15.964" pid="1975" kind="begin" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Processing."></mrk>

    <log tms="412113209.037" tmt="01/22/2014 20:53:29.037" pid="1975" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared] failed, error = 61"/>

     

    Let’s look at this and then propose a clean method of establishing and consolidating your cluster.

     

    Simply the Bonjour service is having a hard time trying to find you and also qmaster been running ragged trying to mount your Cluster 21D262F0-BF7EC314 storage.

     

    Let's fix it.

     

    Basics for the above with Compressor v4.1 and Qmaster.

    • much has been abstracted from the user to help easy implentation and use. This is good me thinks!
    • avoid ticking every option that is available on each host , such facilities aggravate and add to the complexity of your workflow environment
    • isolate the network subnets to use for all host access, file system paths, communication, diagnosis, monitoring  and finally data transfer (see later)
    • use source objects that will develop segments that will be distributed for processing. A 3 minute clip generally won't segment to cause distribution.
    • review any workflow gains by distributed transcoding: slow node hols up process and additional time to assemble qt segments. A cluster dedicated to an iMac or macpro can often be faster.  (Have several clusters defined and submit accordingly (long , most and short )!!)
    • All elements/objects used in the source and any target folders SHOULD (not must) be mounted and accessible by each Qmaster node.  You can use sym links I recall. For reasons of efficiently and easy of diagnosis.   

     

    So.. I'd propose you try and test your setup as follows .

     

    Try this and start from beginning.  Do your best to perform these work instructions. Try also not to deviate if u can

     

    Simple Architecture Approach:


    • Your main Macbookpro or main work mac (refered to by you as "master") shall be designated the qmasterd controller that services batch submissions AND that also provides transcode services.
    • The other macs ("service or slave nodes) will ONLY perform transcoding services and will NOT accept batch submissions.  The slaves / service nodes  will not be able to send their jobs to your master controller for transcoding for example.

     

    Keep it simple! and please follow these steps.

     

    Step 1: Quiesce your clusters and Qmaster

    1. in Compressor.app v4.1 / Preferences / Shared Computers, stop / disable all hosts (both your macs) from automatic file sharing - tick it OFF (it causes issue u have).. More later
    2. In Compressor.app v4.1 / Preferences / My Computer, stop / disable all hosts (both your macs) stop allowing others to add batches to your host. Slide to OFF
    3. On all hosts, quit or force out compressor.app v4.1
    4. On all hosts (macs) use activity monitor.app or unix ps command to Force Quit ( or kill) any currently running qmasterd task and any compressord tasks if you can.
    5. On all hosts, purge | clean out | delete  the Qmaster and compressor structures. This is documented by several of us on this forum but fundamentally you want to preserve your settings and destination templates and delete the rest.  Do these sub-steps on all hosts where u intend to deploy compressor/Qmaster for your distributed transcode processing
      • a. Navigate to /Users/shared/library/Application Support/ and delete the Compressor folder if it exists. By all means use the osx Finder to put it in the trash or just use the faithfully unix rm command to delete it immediately without serialisation :
        • rm -rf /Users/Shared/Library/Application Support/Compressor
      • b. Navigate to your home directory ~/Library/Application Support/Compressor and move or copy any personalised values to your desktop so we can reinstate them later. Copy these two folders if they exist.
          • Settings
          • Layouts
      • And also copy /move any customised destination templates that u used. These are files ending in ".template"
      • Now using the Finder or unix command delete your ~/Library/Application Support/Compressor folder and all it's objects such:
        • rm -rf  ~/Library/Application Support/Compressor
      • c. Dismount (+E or drag into trash) any shared file systems you have manually or automatically shared between your hosts. (Your two macs) . Turn off any auto mounts you may have setup in login items for Qmaster and your source and target libraries.
      • d. After you have done Steps 1a - 1c on all your hosts ,
        • then RESTART
        • and log back into your hosts
        • attempt to delete the trash on each.

         6. Check the activity monitor and confirm there are no compressord sub tasks running. Qmasterd might be running. That's ok

     

    Step 2: set up your dedicated network for your transcoding cluster .. Important!

     

    In this step you will isolate your cluster network to a dedicated subnet. BTW ,o DNS is needed unless you get exotic with many nodes!

     

    You will:

    • use the macs Wifi network as your primary network for NON TRANSCODING work such as email, internet , iChat, iCloud and bonjour (.local) I'm assuming u have this in place
    • use the Ethernet on your macs as your dedicated Qmaster cluster subnet.

     

    For this procedure will make an IP address range 1.1.1.x subnet and manually assign the IP addresses to each host. Ofcourse you can use a smart DHCP router if u have one or a simple switch and use OSX SERVER.app erver 10.9 ($HK190 , €19) on your MAcbookpro... The later for another time

     

      • a). using system preferences/network on your controller mac("master"), configure the built in Ethernet to manual IP address of 1.1.1.1 Yes yes, dhcp would be nice if we also had a dns for this subnet to identify each host (machine name) however we don't.
        • Leave the subnet default to 255.255.255.0, and router to 1.1.1.1 .. It's not escaping anywhere! ( very private LAN for your cluster! )
      • b) repeat step 2a to set the other "slaves" service node only macs whose built in Ethernet to 1.1.1.2 , 1.1.1.3 and so on
      • c) connect these hosts (macs) ethernets together in a dedicated hub / zoned switch or if only two macs, just use a cat5/cat6 Ethernet cable
      • d) on each host (mac) using system preferences/network check for a Green light against the built in Ethernet
      • e) on each host (mac)system preferences/network to make a new network configuration (so that you can fall back incase of error) :  using system preferences/network make a new network location on each mac
        • - edit the Location listbox, edit and DUPLICATE the current location
        • - select and over type the name and change it to "qmaster work" or some name u like save and close the dialogue
        • - back in sys prefs / network select your new location "qmaster work" (from the location list box and "apply"
        • - now click the gear wheel icon on lower left and Reorder the network interfaces so that the WIfi is top (first) followed by Builtin Ethernet .
        • - click "apply"
        • - confirm Ethernet is still green status
        • Do this on each host (mac) .. The slave/service nodes
      • f) on each host (mac) verify that you can address each mac over you new subnet. There's many ways to do it however do it simply via the /applications/utilities/Terminal.app.
        • From mac #1 whose IP address is 1.1.1.1,
        • Enter:
        • traceroute 1.1.1.2 press return and one line should come back.
        • ping 1.1.1.2 and a continuous lines appear with packets and time is ms. Watch 3-4 then use control+C to stop
        • Do the same to the other nodes you may have such as 1.1.1.3 etc
        • Repeat the above from the other hosts. For example from one of the service (slave) macs say 1.1.1.2, t
        • Test the network path back to your main mac 1.1.1.1: using terminal.app from that slave,
        • Enter:
        • traceroute 1.1.1.1 press return and one line should come back.
        • ping 1.1.1.1 and a continuous lines appear with packets and time is ms. Watch 3-4 lines then use control+c  to stop
        • At this point you should have a solid network path between your hosts over Ethernet on the subnet 1.1.1.x


    Step 3: mount all filesystems over the Ethernet 1.1.1.x subnet that are to be used for transcoding source (input | read )  and target (output | to be written )

     

    Simplicity is important at this stage to make sure you know what being accessed.  This is one reason for disabling all the automatic compressor settings

     

    You will use the Finder's "Connect to Server" (+k) from each slave (server) node to access the source and target filesystems on your master mac for the transcoding.

     

    These can be saved as favourites in the "Connect to Server" dialogues

     

    Do this:

     

    A) locate the volumes / filesystems and folders on your mac master where your source objects is contained. Do the same for where the final distribution transcode is to be written with you user access.. "Steffen"

     

    B) On each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs

    • mount the Source folder. Finder / Go / Connect to Server  or +K
    • enter "Steffen@1.1.1.1//Users/steffen/movies/my-fcpx-masters. ( choose you source directory path) .
    • Click connect & password and use the "+" sign to save as favourite

     

    - mount the target folder. Finder / Go / Connect to Server  +k

    - enter "Steffen@1.1.1.1/users/movies/my-fcpx-transcodes. ( choose your target directory path) . Click connect n password and use the "+" sign to save as favourite

     

    Do these for all your slave macs.  Remember you are specifying file paths over the 1.1.1.x subnet

     

    Obviously make sure your slaves have read and write access. Yes and you could also mark these folders and Shared in Finder info then everyone can see them ... your choice

     

    C) verify your access: on each slave use the finder to create a ftest older in those recently mounted volumes. Delete the test folder.

     

    So now all your networks and workflow folders are all mounted and accessible by your slave hosts and it's user

     

    step 4: Set up Compressor v4.1 and Qmaster

     

    Care is initially required here NOT to click needlessly on options.

     

    Recall that you purged most of the compressor.app v4.1 state information on step 1. ?

     

    Compressor.app v4.1 will appear new when u start it. 

     

    1. on the master mac 1.1.1.1 , launch compressor.app v4.1
    2. open compressor v4.1 preferences (command+,)
    3. using compressor.app V4.1 preferences:
      • My Computers tab: turn OFF . Don't allow others to process on this computer
      • Share Computers tab: UNTICK (disable) automatically share files.
      • Advanced tab: WHen Sharing My Computer listbox: select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble
      • Do not specify additional instances yet! Less to troubleshoot
    4. On each slave mac 1.1.1.2 -1.1.1.x
      • launch compressor.app v4.1
      • open compressor v4.1 preferences (command+,)
      • using compressor.app preferences:
      • My Computers tab: turn ON (yes ON) to allow others to process their stuff on this slave computer
      • Share Computers tab: UNTICK (disable) automatically share file
      • Advanced tab: WHen Sharing My Computer listbox select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble
      • Do not specify additional instances yet! Less to troubleshoot !
    5. On the master mac, 1.1.1.1
      • using Compressor.app v4.1, select destinations and add a new CUSTOM destination and navigate the dialogue to your target folder you specified in step 3b ~/movies/my-fcpx-transcodes as an example.
      • Use this custom destination on the batch later
      • in Compressor.app V4.1 preferences/Shared Computers, click the plus "+" sign in bottom left cornet to create a new cluster called "unnamed ".
      • - click in the name and change the name to "Steffenscluster" (I'm not connected to my network as I do this..)

      • - tick on the slaves 1.1.1.2 to 1.1.1.x to add the to your new cluster .. assumethee are on the right in a list.
    6. Your cluster "Steffenscluster" is now active!
    7. Care care! One more thing to do. You SHOULD make the cluster storage for master 1.1.1.1 available to all your slaves. This is important for this test !!
    8. On the master 1.1.1.1, Use the finder to verify that you have these directories built from compressor.app v4.1
      • /Users/Shared/Library/Application Support/Compressor
      • and your own home directory: ~/Library/Application Support/Compressor
      • Dig deeper for the /storage folder in each to see the hexadecimal named folder that represents this cluster "Steffencluster"!
      • These should be manully mounted on each slave. Recall we DISABLED automatic file sharing.
    9. on each slave mac 1.1.1.2 - 1.1.1.x, mount the masters cluster storage file systems. Do this for verifying access of each cluster Slave
      • on each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs
      • mount Qmaster cluster storage folder 
        • Use Finder / Go / Connect to Server  or +k
        • enter "steffen@1.1.1.1/Users/Shared/Library/Application Support/Compressor/Storage
        • Click connect & password and use the "+" sign to save as favourite
      • - mount users Qmaster cluster storage folder 
        • Use Finder / Go / Connect to Server  or command+k
        • enter "steffen@1.1.1.1/Users/steffen/Library/Application Support/Compressor/Storage
        • Click connect & password and use the "+" sign to save as favourite
    10. This you may have 4 new network volumes (file systems) mounted on each mac slave over your dedicated 1.1.1.x subnet!

     

    Step5: submit a job

    1. On master mac 1.1.1.1 launch the new compressor.app v4.1 "Network Encoder Monitor " .. Use +E . new in COmpressor.app V4.1

      • You should see all your nodes all cluster service points for your master and slaves.  Heres you see just one (this macbookair!)
      • On each host (mac) Launch the activity monitor.app and filter compressord .. There they are on each host! 
    2. Nearly there.  
      • On the mac that's the master 1.1.1.1 (controller )
      • Submit a job:
      • Use finder to move some footage etc that more than 15 mins for example into your source directory folder from step 3b (eg ~/movies/my-fcpx-masters.)
      • In compressor.app v4.1 Add this +I to the batch
      • Drag your custom destination on to the batch
      • Set your encoding setting (apple devices best)

      • Open the inspector in compressor.app and select "video" and make sure MULTIPASS is checked. Then change to frame controls at your leisure.  Better means slower
      • Select the General tab and make sure JOB SEGMENTING is ticked!

    3. Now cross fingers and submit it (or +B)

     

    Step 6: Monitoring the Workflow Progress

     

    In compressor.app v4.1 Use the active tab to watch the progress

    1. Open the disclosure triangle and see the segments
    2. Unfortunately u can really see which node is processing. (No more share monitor .. btw for those who care.. thats buried now in /Applications/Compressor.app/Contents/PlugIns/Compressor/CompressorKit.bundle/Contents/Embedde dApps/Share Monitor.app/Contents/MacOS/Share Monitor
    3. Look at the network encoder monitor (cmd + E) to see the instances processing your work

     

     

    Lots of small and overdetailed steps here Steffen and its worth working through.

     

    Simply all these things need to be availble to get your cluster to work EVERYTIME.

     

    I might update this and post a more detailed disalogue/transcript on my blog and post it here.

     

    Epilogue:

    I for one rather like the new compressor.app V4.1. Qmaster is buried and works well when not teased or unintentionally fooled.

    I would like the ability to :

    • specify the location of the qmaster cluster storage rather than have it in the root systems file system - I used to have it on my disk array
    • compressor to be applescriptable

     

    Post your results for others to see.

     

    Warwick

    Hong Kong


  • 11. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    Wow, what a novel! But thank you very much for writing this long and very detailed step by step instruction.

     

    To cut a long story short. It worked only once. I did every single step. As expected, the crux was the mounting of the network volumes. Mounting with "steffen@1.1.1.1/Users/steffen/Library/Application Support/Compressor/Storage" failed. I instead added the Storage folders and the source and destination folders manually in the Shared preference pane of Mac OS X and mounted the folders with "steffen@1.1.1.1/". Up came a dialog where I could select all four shared folders. There was no storage folder in /Users/shared/Library/Application Support/Compressor/ so I added it manually.

     

    The job was done successfully. I tried another job with the same settings. The master Mac created 4 segments and worked one after another for itself. There was no distribution to the slave Mac anymore. I did the whole setup procedure again with no success. I deleted the folders, made many restarts. Nothing. The slave Mac again gives the error message in the Network Encoder Monitor that it could not find the source file. I am so tired of this Compressor clustering thing.

  • 12. Re: How to share a job with Compressor 4.1?
    Warwick Teale Level 3 Level 3 (575 points)

    Hi Steffen, well it's better having a deal of detail than advice that's based on the drone of "removing prefs" and "repairing disk permissions" based on some but not all "folk lore" and perhaps myths!

     

    (Sad that the screen snapshots did not consolidate into my previous post.. my error)

     

    To your latest effort. It certainly looks like you had some pleasure with this, although it was brief.

     

    I am certain that this latest issue is simply related to the ACCESS and the availability of the objects (resources) you need for your transcoding transactions.

     

    Steffen Bendix wrote:

     

    .....snip snip. It worked only once. I did every single step. As expected, the crux was the mounting of the network volumes. Mounting with "steffen@1.1.1.1/Users/steffen/Library/Application Support/Compressor/Storage" failed. I instead added the Storage folders and the source and destination folders manually in the Shared preference pane of Mac OS X and mounted the folders with "steffen@1.1.1.1/". Up came a dialog where I could select all four shared folders. There was no storage folder in /Users/shared/Library/Application Support/Compressor/ so I added it manually.

     

    This may be true because initially  qmaster has likely utilised /users/shared/application support/compressor/storage/xxxxx. It certainly does not hurt anything if you add this structure with the correct access and permissions.

     

    Steffen Bendix wrote:

     

    The slave Mac again gives the error message in the Network Encoder Monitor that it could not find the source file.

    Yes, then there is certainly an access issue here that's likley very very easy to fix. THe detail of the resource missing will be in the host SLAVE (1.1.1.2) /users/steffen/library/logs/compressor/ check these logs out ... especially the com.apple.stomp.transcoderx. logs  use the console.app .

    console_compressor.png

    In the log above you can see can see the successful access of the host. In you case, I'd be certain there was an error in eht slave for this.

     

    This is merely a file path or access to the source r the target.

     

    worth checking. Post here if you like.

     

    When you get your system stable, THEN bugger around with instances and some automation.

     

    I use shared dedicated directories for the client jobs I do. This way I have everything accessible from each host. I also use destination templates which is a huge workflow help.

     

     

    Steffen Bendix wrote:

     

    . I am so tired of this Compressor clustering thing.

    Yeah I can imagine

     

    Stick with it though mate. It works very very well when its sorted out.

     

    I believe Apple's intention was to make it very very easy to use. They have done a good job so far with compressor.app v4.1 (2013)

     

    I think the available Compressor.app V4.1 documentation is very light and over simplified for resolving issues. It's compounded by urban myths and legends and some good intentioned "delete the prefs" people without the time or perhaps the need to look into the problem.

     

    FWIW: I am asked to help out clients with this so I keep on top of it as best as I am able such as Russ and a few others.

     

    Keep trying Steffen and I beleive you will be not only happy when you get it all working, and additionally you will make SOME of your workflow more efficient and become more knowledgeable in the process.

     

    Post your results for others to see.

     

    Warwick

    Hong Kong

  • 13. Re: How to share a job with Compressor 4.1?
    Steffen Bendix Level 1 Level 1 (40 points)

    Warwick, you are very helpful. I repeated the setup procedure again with no avail. Here are the error messages from slave's transcoderx.log.

     

    <mrk tms="412286584.450" tmt="01/24/2014 21:03:04.450" pid="2123" kind="begin" what="log-session"/>

    <mrk tms="412286609.127" tmt="01/24/2014 21:03:29.127" pid="2123" kind="begin" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg="Processing."></mrk>

    <log tms="412286622.155" tmt="01/24/2014 21:03:42.155" pid="2123" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35/shared] failed, error = 61"/>

    <mrk tms="412286629.224" tmt="01/24/2014 21:03:49.224" pid="2123" kind="end" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg=" error: Aktivierung von Shared-Storage fehlgeschlagen: Host = Idefix.local, Pfad = /Users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35/shared"></mrk>

    <mrk tms="412286712.936" tmt="01/24/2014 21:05:12.936" pid="2123" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412286722.593" tmt="01/24/2014 21:05:22.593" pid="2123" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg="Error: unrecognized request."></mrk>

     

     

    After this I mounted the folder "64DE…" directly and go this:

     

    <mrk tms="412289024.344" tmt="01/24/2014 21:43:44.344" pid="2123" what="service-request" req-id="A84C4AF0-7542-4ED9-A5B3-CA91901EEBCF:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412289065.206" tmt="01/24/2014 21:44:25.206" pid="2123" kind="begin" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg="Processing."></mrk>

    <mrk tms="412289070.244" tmt="01/24/2014 21:44:30.244" pid="2123" kind="end" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg=" error: Ausgangsdatei „/Users/steffen/Downloads/Compressor/Source/Test.mov“ wurde nicht gefunden"></mrk>

     

    The last line in German means that the source file was not found. This is nonsense because the folder "Source" is mounted and has read/write access from the slave computer.

  • 14. Re: How to share a job with Compressor 4.1?
    Warwick Teale Level 3 Level 3 (575 points)

    HI Steffen, yes you have seen these MOUNT failures in the compressor stomp logs.. let's have a quick look...:

    Steffen Bendix wrote:

     

    <mrk tms="412286609.127" tmt="01/24/2014 21:03:29.127" pid="2123" kind="begin" what="service-request" req-id="C374AD45-B1C9-4E4A-897D-9645C1121BF2:1" msg="Processing."></mrk>

    <log tms="412286622.155" tmt="01/24/2014 21:03:42.155" pid="2123" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35/shared] failed, error = 61"/>

     

     

    The first failure  that you have documented here in Qmaster on your slave where it seems to be trying to mount its cluster storage for your cluster  whose internal (global) name is ) 64DE4D86-4736EB35  on the MASTER  that I recall should be IP address 1.1.1.1 ? (y/n)?

     

    Instead its trying to mount your cluster storage at /users/steffen/Library/Application Support/Compressor/Storage/64DE4D86-4736EB35 / ...... that it believes is on the local host (Slave @ 128.0.0.53?).. should not be.?

     

    Recall you have set automatic file sharing to OFF in the compressor qmaster cluster setup. This helps to get this working and understand what is going on here

     

    By the way, did you have a dedicated path set to BUILTIN ETHERnet on the master (@ 1.1.1.1) and the slave ( @ 1.1.1.2) in the compressor cluster (group) parms:? please check else it will take any path and that might work however you want to push these transactions over the 1.1.1.x subnet me thinks!

     

    Here's an example:


    compressor_NIC.png

      • please disregard my use of the "Enable instances" and the "Enable Port range" .. leave them off for the moment. .. I just logged into one of my servers to get this.. I'm on the road (Chinese New Year this week!)
      • this one of the NIC's (network Interface 'Cards"/Channels) from a mac mini in my transcoding cluster (Thunderbolt - Ethernet adapter over a path 10.0.2.x/16, OSX server DNS and DHCP).

     

    Verify that you have specified "Built-in Ethernet" or its name on ALL (Both) of your machines for the subnet 1.1.1.x.

     

    Series 01 Questions and Actions to try:

    1. do you have a dedicated network path from your slave (1.1.1.2) to you other mac at 1.1.1.1? (should be ok, please verify from master IP @ 1.1.1.1). (As above)...
      • try this from the slave ( @ 1.1.1.2) using Terminal command: traceroute -s 1.1.1.2  1.1.1.1 (us ethe -s switch to and the 1.1.1.2 parameter to PUSH the traceroute from the slave's 1.1.1.2 interface .. your ethernet I recall).. should get one line that looks ok. IF a no-route or similar, please  set up your ethernet point to point. should be a no brainer
    2. from your slave machine (1.1.1.2) can you SUCCESSFULLY NETWORK mount your private cluster storage on to you slave at 1.1.1.2? (note that resides on your master mac (1.1.1.1)  at paths:
      • /users/steffen/Library/Application Support/Compressor/Storage
        • use Finder / go / "connect to server" or +K - enter as text /users/steffen/Library/Application Support/Compressor/Storage and "connect" - put your logonid and or password in as/if requested in the AFP dialogue. The file system should mount on your slave mac (see the desktop).
      • /users/Shared/Library/Application Support/Compressor/Storage (if it exists) else bypass the next sub-step.
        • use Finder / go / "connect to server" or +K - enter as text /Users/Shared/Library/Application Support/Compressor/Storage and "Connect" - put your userid and or password in if | as requested in the afp dialogue. The file system should mount on your slave mac (see the desktop).
      • you probably have two filesystems mounted on your slave called "Share"... open them and see if you cna add and delete a file or folder.
      • recall in the previous posts I suggested you make a favourite of these in the "connect to server" dialogue.
      • FWIW - when this settles down it will be possible ot have these automatically mounted.. Rather than let it default, I just have a rudimentary Automator.app workflow .. click and forget.. then it's completed.. this later if you're interested

     

    so far so good? - (... over..)

     

     

    Steffen Bendix wrote:

     

      After this I mounted the folder "64DE…" directly and go this:

     

    <mrk tms="412289024.344" tmt="01/24/2014 21:43:44.344" pid="2123" what="service-request" req-id="A84C4AF0-7542-4ED9-A5B3-CA91901EEBCF:1" msg="Error: unrecognized request."></mrk>

    <mrk tms="412289065.206" tmt="01/24/2014 21:44:25.206" pid="2123" kind="begin" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg="Processing."></mrk>

    <mrk tms="412289070.244" tmt="01/24/2014 21:44:30.244" pid="2123" kind="end" what="service-request" req-id="2F920273-FFB5-4FBC-BD8B-784E2C5DF3C4:1" msg=" error: Ausgangsdatei „/Users/steffen/Downloads/Compressor/Source/Test.mov“ wurde nicht gefunden"></mrk>

     

    The last line in German means that the source file was not found. This is nonsense because the folder "Source" is mounted and has read/write access from the slave computer.

     

    Thanks for the translation BTW mate!..

     

    Series 02 Questions and Work Instructions.:

    1. is the file system where your source (I assume its a source .. good name anyway!) for "Test.mov" is MOUNTED on the SLAVE system (@ 1.1.1.2) ? It's complaining about it that the qmaster prefight (setup) can't find the path mentioned above to your source(Test .mov) on your master @ 1.1.1.1
    2. . To be precise and simplistic..
      • confirm for this post that the file path in the error message is actually ar Master (@ 1.1.1.1)  @1.1.1.1/Users/Downloads/Source/Test.mov?
        • yeah I know.. I'm just confirming with you.
      • confirm that you have this file path (Folder is "Source") mounted on your slave mac  @ 1.1.1.2) from your Master @ 1.1.1.1 ?
      • have a look from slave @ 1.1.1.2
        • via Terminal as: ls -1 /Volumes
        • via Terminal as df
        • in Finder under Shared sidebar and on the desktop?
    3. IF not then from the slave @ 1.1.1.2,  MOUNT the file path where your source is :
      • Finder / Go / "Connect To Server" or +K and add the path as :
        • afp://steffen@1.1.1.1/users/steffen/Downloads/Compressor/Source
        • make a favourite in the dialogue for your convenience.
    4. (BTW.. when you sort this out, you can utilise some simple setup in Automator.app so that you donthave t bugger wi this evry time.. it's excellent!)
      • (later , use a set up known MAster directories, and Client distribution directories and make them into TEmplates (destinations etc) and it will always work. .. later for another thread ).

     

    IF you have tried the above, do you still get this error that you reported above?

     

    Do you get the same error for the 1.1.1.x path? If so check for access (+i) on the cluster. free free to add an ACE (access control entry) the the ACL (access control list) and give yourself steffen read+write and access if you don't have it for some bizarre reason!

     

    In case you do:

    Do this in the Master @ 1.1.1.1 where you Source resources are:

    ace-steffen.png

     

    (You should NOT have to do this.. where you see my name it should already have steffen as owner and read write).

     

    Submit your batch again.

     

    Summary:

    To make this work all around using a distributed node, it's a good idea to have ALL the source elements required  and target output paths and the cluster all shared to all hosts on the cluster (you @ 1.1.1.2 slave mac) as I remarked in my earlier posts.

     

    One would anticipate that all tihings being transparant and easy that Qmaster / compressor would attend to all tis. It does when you have evertything available. In this case we have purposely NOT used automatic file sharing... you can see why doing this MIGHT cause issues of performanace , access and diagnosis.. this is why I dont utilise it. INstead I manaual maount all my source and targe resources.

     

    Others reading this post  may critique my procedure and note it unnecessary... and this is to offer a fair opinion that QMaster should take care of all this by copying resources to the shared cluster (spool) and permitting nodes to access from there!. I say "Yes", to those opinions...  Qmaster may do this, however it's time consuming and seems to be error prone at times, hence this manual approach used my others and myself.

     

    Give it another go mate! You're nearly there.

     

    Post your results back for others to see.


    Warwick

    Hong Kong

1 2 3 Previous Next