How to share a job with Compressor 4.1?

HI Steffen looks good! THe error you describe is my doing. I my haste I typed in the INCORRECT specification (typo) for your path as

(@ 1.1.1.1) @1.1.1.1/Users/Downloads/Source/Test.mov?

It should have been:

Master (@ 1.1.1.1) steffen@1.1.1.1/Users/steffen/Downloads/Source/ 😢

Steffen Bendix wrote:

I checked everything and every volume is mounted. There is read/write access from the slave. The Macs are connected over direct Ethernet. "Ethernet only" is set in the preferences of both Compressor.apps. To avoid conflicts I disabled WIFI on the slave Mac. Here is the output of the terminal.

Friedward:~ steffen$ ls -l /Volumes

total 8

drwxrwxr-t 38 root admin 1360 1 Dez 13:41 Backup

drwx------ 1 steffen staff 264 26 Jan 09:09 Finished

lrwxr-xr-x 1 root admin 1 26 Jan 08:54 Friedward -> /

drwxrwxrwx 0 root wheel 0 26 Jan 09:07 MobileBackups

drwx------ 1 steffen staff 264 24 Jan 21:37 Source

drwx------ 1 steffen staff 264 24 Jan 22:04 Storage

drwx------ 1 steffen staff 264 24 Jan 22:04 Storage-1

Friedward:~ steffen$ df

Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on

/dev/disk0s2 877175552 381377104 495286448 44% 47736136 61910806 44% /

devfs 365 365 0 100% 632 0 100% /dev

/dev/disk0s4 97656248 94704872 2951376 97% 11838107 368922 97% /Volumes/Backup

map -hosts 0 0 0 100% 0 0 100% /net

map auto_home 0 0 0 100% 0 0 100% /home

localhost:/NgXXEf4A4YasM2yTNDBPSd 877175552 877175552 0 100% 0 0 100% /Volumes/MobileBackups

//steffen@1.1.1.1/Source 1654970944 1072194680 582776264 65% 134024333 72847033 65% /Volumes/Source

//steffen@1.1.1.1/Finished 1654970944 1072194680 582776264 65% 134024333 72847033 65% /Volumes/Finished

//steffen@1.1.1.1/Storage 1654970944 1072194680 582776264 65% 134024333 72847033 65% /Volumes/Storage

//steffen@1.1.1.1/Storage-1 1654970944 1072194680 582776264 65% 134024333 72847033 65% /Volumes/Storage-1

One of the Storage folders is automatically titled "Storage-1" to avoid names conflicts I think. May this be the reason?

From the slave over Ethernet 1.1.1.2 this looks pretty good. Yes you are right you will have Storage-1 and Storage to resolve the duplicate names. You certain can name these properly to avoid confusion... however let's do this later.

I can see drwx------ 1 steffen staff 264 24 Jan 21:37 Source is mounted. So your own mounts worked well!

For our discussion, the file system "Finished" is the TARGET of the transcode? IF yes, then it looks like you have performed admirably 😉 and it all should work because every thing looks in place and the slave (1.1.1.2) may surely access these file systems in read/write mode (rw) from your listing.

Steffen Bendix wrote:

On either Mac I can't enable any instances.

Instances of the compressord subtask: is this to which you refer?

If you can't ENABLE ADDITIONAL instances it nor SELECT (form the listbox) any more instances, it's quite normal.

REASON: I understand it's a NEW feature of Qmaster/Compressor for thoughput balancing. Its seems that this part of the UI examines the computer resources you have for host to base this for further additions. This is the main reason I proposed you DON'T bother to set it on whilst you are still undertaking to stabilise your new cluster.

FWIW, grand central dispatch will put you available to cores to work... Suggest you dont be concerned.

So, have a look at what's running now on your slave and your master:

Can you see some compressd subtasks running on your slave & your master?

USe the Activity Monitor.app to have a look. Use the filter compressord in to locate only those.

As an example on our MAC PRO nehalem with an ATI 5870 graphics card, 16 virtual cores and 32GB of ram, the new Compressor.app 4.1 UI qmaster will only permit 3 additional instances. The 2011 mac minis are only permitted ONE additional instance (16GB ram i7)... These run flat out maximum with qmaster.

Lastly, were you able to submit a batch form your master using your CLUSTER?

IF not, would you post any errors you see in ~/logs/compressor (console.app)

You have it all working soon enough mate!

Post your results for others to see.

Warwick

Hong Kong

HI Steffen, thanks for posting your screen views. I think you are measuring / monitoring using the wrong (our out of date tool). I think also you stuff works and you may be looking in the wrong place to see it working 😐. I also don't thing you have been sumitting the jobs on your own defined cluster.

LEt's see.... FWIW you've done all the correct things to get this going. I think you'll kick yourself when you see... ,

My Deutsch is worse than dreadful.., however does "Deiser Computer" mean "This Computer" in English? If so then I think you are looking at the wrong thing mate!😀.

What gives it away is that the icon (looks like a pee-sea) is not an icon for a defined cluster for distributed nodes cluster 😉

Some Observations:

1. The utility Share Monitor.app is hidden away (buried) these days in Compressor.app 4.1. I noticed myself that it now WONT SEE the clusters (groups) defined in Compressor.app v4.1. I will stand corrected on this though.
2. Note yourself that SHARE MONITOR.app is showing the job running on Deiser Computer/This computer.. This is NOT your defined/created CLUSTER name (aka "Group" ,them now used in Compressor.app V4.1) .. therefore the job will not utilise any distibuted nodes.

   I'll be pleasantly surprised if it is..! Please verify for my curiosity.

3. The Compressor.app V4.1 UI "Active" certainly shows segments transcoding on your envirorment...
4. and lastly you've confirmed you and instance of compressord running on it.

Here are screenshots of the master and the slave computer while computing:

Detail:

The other screen shots of Compressor.app V4.1 UI on both the MAster (1.1.1.1) and the Slave (1.1.1.2) certinly look correct as you have pointed out.

SO.. Im very very curious if you have:

1. successfully defined a CLUSTER (group) that is a UNION of the master (1.1.1.1) and slave (1.1.1.2) and named it something like "steffencluster"?
   • if that was true, AND Share Monitor.app (that is now buried away) was correctly showing the cluster 'steffencluster" and not what is "Diesier Computer/This Computer"
   • 
   • the above if from Page 2) of the new  Apple Compressor V4.1 USers Guide (Dec 2013).
2. SUMBITTED the "Test.mov" job using the group defined (cluster)?
   • you MUST submit the job 'tets.movie" and its distibutions (u may use more than one) , to a GROUP or CLUSTER that represents the distirbution nodes (1.1.1.1 and 1.1.1.2 etc)
   • here is an example (copied from PAge 102 of the  Apple Compressor V4.1 USers Guide (Dec 2013).
   • Looking at Compressor.app V4.1. UI in the Current Tab where you set up the job. look down at the middle lower left for a "process on: list box.
   • Do you see you "steffencluster" in the list? IF so , select it and use THAT "steffencluster" for your distributed processing and not "Deiser Computer" (This Computer)?

DId you submit your 'TEst'.mov" using the CLUSTER you defined???.. it doesn't look like it mate 😕 !

Please verify and post back. I'm rather curious...

Post your results for others to see.

Warwick

Hong Kong

Hi steffen, please refer to my previous post.

I believe you're nearly 99.99% there to getting this all working.

When you do you will be very pleeeeaaaaaased!

You will appreciate that Compressor.app is your friend again.

Warwick

Hong Kong

HI Steffen,

Steffen Bendix wrote:

yes, I see my cluster in the preferences but in the activity section of Compressor and in Share Monitor there are only the entries "This Computer" ("Dieser Computer" in German). The once and only time sharing worked some days ago, Share Monitor successfully showed which Computer worked on which chunk but there was never any other entry than "This Computer" in the list.

Share Monitor.app..

• hmm I porpose you don't rely / use Share MOnitor.app for Compressor.app V4.1. 2013.
• From our experience, it has been ("purposely buried' - like in the movie 2001!)
• There is a NEW item Network Encoding Monitor.app (command+E , +E) that is barebone and shows you the instances that are active. I mentioned this in a previous post. USe the "SHOW LOGS" to watch the STOMPxxx compressord status progress
• Share Monitor.app at its best was often unreliable.. (socket errors) suggest you don't "bank" on its improvement especially that it's been moved out of the /Applications folder to oblivion in a Contents somewhere i mentioned on these forums LoL!
• The ACTIVE pane in Compressor.app V4.1 UI is your best bet. It currently lacks showing where the job is being dispatched on what nodes. PErsonally I think this is part of simplifying it for everyones use.
  • also using the Activity Monitor.app and
  • also the ~/Library/Logs/Compressor/ as I mention above
  • these will show you if you segments have been dispatched on the 1.1.1.2 slave node.

I found out something more yesterday. When I start the job with the test.mov lying on the SLAVE computer, the log on the slave shows the error that this file cannot be found at "/Volumes/Users/steffen/Downloads/Compressor/Source/Test.mov". When the mov-file is situated on the MASTER computer, the error reads "/Users/steffen/Downloads/Compressor/Source/Test.mov. Compressor on the slave searches for the file always in the wrong location.

Yes this is very strange, seeing that these are mounted, THis is certinaly the error tha twill stop the SLAVE 1.1.1.2 from having those segments dispatched on them.

I dunt know what else I can suggest here. Unless it's some envirnment steffen-only circumstance, it shoudl be able to be resolved using some of the methodology we and have have offered to you.

Regarding the above would you verify that the AFP failure (access by the cluster) is for 1.1.1.2/volumes/Source/ that should be the mount point it shoudl be accessing.. not the /users/steffen/downloads/compressor path... seems strange that the slave would ask fo that.

Steffen Bendix wrote:

And here is something more: I created a new user on the slave computer. I can access the 1.1.1.1 network without any problems but I can't see the slave in Compressors clustering preferences. The slave doesn't see the master, too. I don't have the firewall or any other similar software running which could prevent the network connection.

Interesting... fwiw

I would have thought the cluster (steffenscluster" was global especially when that host 1.1.1.2 is part of it.. I dont know.

Slave: recall you shoudl have "Allow others to process.." = OFF on that slave 1.1.1.2 host, thus that SLAVE cannot submit job to the 1.1.1.1 master. MAybe a reason.

Additional docs:

Steffen, I believe you are already using this correctly and the failures in teh set up are resolvable.

Would you be so kind as ather some screen shots (shift+cntl+command+4), then Preview.app and +N , make and crop a .PNG's and upload to this thread for:

On the master 1.1.1.1

1. Compressor.app V4.1 Preferences / Shared Computers
2. Failed part of any STOMP and CONTROLLER logs for one job in your ~/Library/Logs/Compressor
3. a grab of the Compressor.app V4.1 Current pane with all the options you have selected to SUBMIT the "test.mov' Batch so we can see what you are submitting
4. NEtwork Encoder Monitor - a screen grab of that - there should be two instances as IDEL (one on each node 1.1.1.1 and 1.1.1.2)

On the slave/service node 1.1.1.2

1. Compressor.app V4.1 Preferences / Shared Computers
2. relevent Failed part of any STOMP and CONTROLLER logs for one job in your ~/Library/Logs/Compressor
3. Network Encoder Monitor - a screen grab of that - there should be two instances as IDEL (one on each node 1.1.1.1 and 1.1.1.2)

See how you go mate!

Warwick

Hong Kong

Hi Steffen, thanks for the logs and screen images:

Firstly, i WILL STAND CORRECTED that Share Monitor.app DOES show a cluster correctly. I simulated my setup something like yours and launched the setup, and to my surprise (and self-admonishment!) .... in fact you can see Share Monitor.app showing the DISTRIBUTED nodes (node in this case) running🙂. The unusual (as per Compressor V4.0) it shows the active cluster as THIS COMPUTER which I regret telling you it was an error.. 😢

You can see in this job below that compressor.app v4.1 has divided this job into 4 segments across the cluster:

• the main node (mac pro)
• and the slave (the macmini).
• The macpro is the CONTROLLER (can submit jobs). click to enlarge.

Steffen Bendix wrote:

I dunt know what else I can suggest here. Unless it's some envirnment steffen-only circumstance, it shoudl be able to be resolved using some of the methodology we and have have offered to you.

I tested it with three different computers in any combination. It is the same error on all of them.

YES this is quite perplexing indeed. Its looking simply likely that the resources ARE NOT really available to all the cluster. see later..

Regarding the above would you verify that the AFP failure (access by the cluster) is for 1.1.1.2/volumes/Source/ that should be the mount point it shoudl be accessing.. not the /users/steffen/downloads/compressor path... seems strange that the slave would ask fo that.

I don't really understand. Do you mean I should check that the network volumes (Storage folders etc.) can be accessed by the slave computer? Yes they can.

FILE SHARING:Well something YOU could ADD just to try out is to specify the folder as being able to be shared for all. I'm assuming you don't have OSX SERVER running. Try this in the Finder (it won't hurt anything).. like this example (click to enlarge)

On each HOST (1.1.1.1 and 1.1.1.2) for each directory (folder) that you want to share in the cluster (between these two hosts including the /compressor/storage etc)

1. Using the Finder, select the directory
2. use File / Get Info (+i)
3. then Tick (enable) Sharing
4. the resulting SHARENAME (Sharepoint) will be the name of the folder that the hosts can see.
5. do this for ALL folders on all hosts used by the cluster (you can turn it off later)

➕ Re do again the folder mounts of the master 1.1.1.1 filesystems you need.. do this form the Slave Finder:

Now...on the slave (1.1.1.2) dismount (eject | +e) and remount / share them as you have been doing. In fact this time try this in the Finder to remount:

1. On Slave (1.1.1.2) in Finder
2. Connect to Server (+k)
3. enter 1.1.1.1 as the host (because we want them mounted over this specific ethernet path)
4. you will see a list of SHARED mount points (these appear now because of your actions above!)
5. select the ones you need accessible by the host 1.1.1.2 (current host) for the cluster.

Try your batch job again. Recall you can always return and disable these settings.

Further...

Slave: recall you shoudl have "Allow others to process.." = OFF on that slave 1.1.1.2 host, thus that SLAVE cannot submit job to the 1.1.1.1 master. MAybe a reason.

You mean the master should have switched OFF "Allow others to process" and the slave switched it ON? Otherwise the master could not submit anything. And yes, the settings are like that.

YES you are correct!: Whoops another iPHONE typo 😠.. sorry once again the SLAVE must be ON - , while you are testing this, there is no need to have the MASTER (1.1.1.1) must be OFF for our testing, be the only one also be a method to submit jobs and its OWN. I pointed out originally I have suggested this only to mitigate (reduce) any other logs and problem errors. (As you have done)

Here's is the example of my simulated setup for you. Here you can see this mac-mini (service node) Compressor.app V4.1 preferences/My Computer (Deiser computer)

The rest of the setup looks correct.

• the job has JOB SEGMENTING enabled
• and the job is performing a SEGMENTED TRANSCODE on the iMAC (1.1.1.1 master) ..
• settigs look good...

3. Compressor options iMac:

So.. the issue of course is that the slave (1.1.1.22 - the MBpro) cant access the resources and thus will not become engaged in the cluster for this job.

Note that the slave node (1.1.1.2 - MPB) CANNOT see the cluster you defined! it will be "Untitled" as a default.( you can rename it on the slave)

here's is the Compressor.app V4.1 Preferences/Shared Computers from MY SIMULATION setup for the SLAVE (mac-mini-01)

⚠ and here is the one BELOW from your MAcBook Pro (1.1.1.2 - slave) - I dont see the cluster?

Looking at the MPros compressor.STOMP log below in RED there is ofcourse the error that the path to an object (some resource - your Test.mov) wurde nicht gefunden (not found??) can be found on the current host (this host) and thus this node (1.1.1.2) is excluded from joining in the cluster for this job (839C7087-B6A2-4302-A505-70D020186B4B)

2. Log of transcoderx MBP:

<mrk tms="412584903.688" tmt="01/28/2014 07:55:03.688" pid="18591" kind="begin" what="service-request" req-id="839C7087-B6A2-4302-A505-70D020186B4B:1" msg="Processing."></mrk>

<mrk tms="412584903.742" tmt="01/28/2014 07:55:03.742" pid="18591" kind="end" what="service-request" req-id="839C7087-B6A2-4302-A505-70D020186B4B:1" msg=" error: Ausgangsdatei „/Users/steffen/Downloads/Compressor/Source/Test.mov“ wurde nicht gefunden"></mrk>

<mrk tms="412584904.689" tmt="01/28/2014 07:55:04.689" pid="18591" what="service-request" req-id="839C7087-B6A2-4302-A505-70D020186B4B:1" msg="Error: unrecognized request."></mrk>

<mrk tms="412584989.550" tmt="01/28/2014 07:56:29.550" pid="18591" what="service-request" req-id="839C7087-B6A2-4302-A505-70D020186B4B:1" msg="Error: unrecognized request."></mrk>

<mrk tms="412585019.952" tmt="01/28/2014 07:56:59.952" pid="18591" what="service-request" req-id="839C7087-B6A2-4302-A505-70D020186B4B:1" msg="Error: unrecognized request."></mrk>

<mrk tms="412585165.979" tmt="01/28/2014 07:59:25.979" pid="23199" kind="begin" what="log-session"/>

<mrk tms="412585288.583" tmt="01/28/2014 08:01:28.583" pid="23199" kind="begin" what="service-request" req-id="6416F3AF-D362-40F6-A9B4-2F9C0903D0DF:1" msg="Processing."></mrk>

<mrk tms="412585293.626" tmt="01/28/2014 08:01:33.626" pid="23199" kind="end" what="service-request" req-id="6416F3AF-D362-40F6-A9B4-2F9C0903D0DF:1" msg=" error: Source file /Users/steffen/Downloads/Compressor/Source/Test.mov not found"></mrk>

<mrk tms="412585322.120" tmt="01/28/2014 08:02:02.120" pid="23199" what="service-request" req-id="6416F3AF-D362-40F6-A9B4-2F9C0903D0DF:1" msg="Error: unrecognized request."></mrk>

and .. as the English (the Poms say) ..'the plot thickens!" 😉

Additionally the Network Encoder Monitor on the MACBOOKPRO (1.1.1.2) DOES NOT see the iMAC instances - looks bad.--

3. Network Encoder Monitor idle MBP:

ANd the NETWORK Encodong Monitor is showing the RED message confirming the resources was not found. (last post from steffen https://discussions.apple.com/content/attachment/370535040)

What to DO?:

As in a few posts ago, I thought you had this mailed down pretty much.

1. It looks to me as though DIRECTORIES you need from 1.1.1.1 (master) and simply NOT MOUNTED / SHARED on slave (1.1.1.2 - MACBOOKPRO)
2. On the SLAVE nodes (1.1.1.2) Try the "SET THE SHARING tick on " in the finder as I mentioned above. The FINDER is very easy to do like that.
3. then use the Finder to check eth access and permissions and the path too while you are there (+i).

(FWIW, I use OSX SERVER to administer the complex I have here. You definitely DO NOT need it BTW.

Mate you're so close to getting this working😀.. Please don't give up (Warwick's pep-talk to Steffen!)➕

Post your results for others to see.

Warwick

Hong Kong

Hi Steffen, looks like you have had some success.

Franky ghat suggestion from Apple as well meaning as it might seem is nonsense. By default if one was to use the ~/Movies directory as the examples Apple has in manuals from the past then their own advice would be incorrect.

I believe you just have a simple setup issue with SHARING your file systems.

Anywy you have it working by the sounds of things.

Good work

W

Hi Steffen, regarding your most recent post:

Steffen Bendix wrote:

.....e but when the last piece is almost finished (approx. 80%) the last computer finishes its work. The final movie can't be completed by the master computer. The individual work pieces remain uncombined.

It's likey that access to the SEGMENTS from one host (the master or the slave) has been removed

If you are able to re-run the job again and check the following ON BOTH Hosts (master and slave 1.1.1.1. and 1.1.1.2):

1. the (console/compressor) ~/logs/compressor/servicecontroller/com.apple.stomp.transcoderxx.logs - there will be an error in here. Might be QT or encoding error.
2. check the system.log too for any errors. Look for things like:

   spinlock smashed, error AFP or NFS mount failure, loss of path,, NIC error (error on ETHERNET ) etc ec

3. also check these places for the assempled segments:
   • /users/shared/Library/application support/compressor/storage/xxxxxx-xxxxx/ for partial segments
   • (your home's) ~/Library/application support/compressor/storage/xxxxxxx-xxxxx/Jobs and look for segments if they exist

The error or resource issue will be in here somewhere.

Post your reults for others to see.

W