Previous 1 2 3 Next 127 Replies Latest reply: Jan 29, 2015 6:42 AM by essandess Branched to a new discussion.
essandess Level 1 Level 1 (5 points)

I use EyeTV HD and Turbo.264 HD to record and transcode h.264 shows for playback on a Mac, Apple TV, iPad, iPhone. I also use a self-modified version of etvcomskip to detect and mark commercials, which are then automatically skipped at playback. This makes tv viewing a lot more pleasurable and efficient, and is especially helpful controlling advertiser access to children. In case this is useful to others, I've outlined the steps here:

 

  1. Go donate $10 or more at Eric Kaashoek's website Comskip and download his HD-capable Comskip software. "Commercial" commercial-skipping companies (ReplayTV, TiVo, Dish, etc.) are quickly sued out of existence or into submission, even if they simply provide a basic "30-second skip" feature. Comskip is open source beerware that you can download, install, and use yourself, and you can't sue anyone for publishing source code. And rather than providing a simple 30-second skip feature, it detects, marks, and provides that capability for automatic commercial skipping.
  2. Comskip a Windows executable, so you'll need WINE (WINdows Emulator) on OS X to run it. First, download and install Xcode (free from the Apple App store), then download and install and install Macports, then in Terminal, run "sudo port selfupdate; sudo port install wine-devel". Finally, make sure that /Applications/Utilities/X11.app is added to your list of applications to run at login (WINE uses X11 to display Windows windows). Yes, this is windows emulation, but on a half-decent box you'll see that commercial skipping runs at around twice real-time for h.264 HD recordings.
  3. Download and install etvcomskip (and PyeTV if you want EyeTV to work with Front Row). The 2010 version of etvcomskip doesn't work with Comskip's latest HD capabilities, and the latest Comskip mpeg libraries do not allow live tv processing, so install these files to get everything working. Once you done this, you'll have a commercial-skipping HD DVR box on your Mac that can stream to Apple TV, iOS, or export to iTunes.

Comskip software

This lives in the directory /Library/Application Support/ETVcomskip.

 

Edit the file ./comskip/comskip.ini to use these HD-tuned settings, tailored for US cable:

 

detect_method=127

;1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all
max_avg_brightness=21

;maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
non_uniformity=50

; Set to 0 to disable cutpoints based on uniform frames
max_volume=20


; any frame with sound volume larger than this will not be regarded as black frame
logo_at_bottom=1

; Set to 1 to search only for logo at the lower half of the video, do not combine with subtitle setting
intelligent_brightness=1
; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA

volume_slip=150

output_dvrmstb=0

; Set to 1 if you're running DVRMS-Toolbox
live_tv=0



; set to 1 if you use parallelprocessing and need the output while recording
live_tv_retries=4

; change to 16 when using live_tv in BTV, used for mpeg PS and TS

 

Edit the file ./MarkCommercials.app/Contents/Resources/MarkCommercials.py to use WINE to run comskip.exe nicely. Grep for these changes:

 

EyeTV.launch()

 

msg = 'Error: unable to communicate with %s\n' % options.app

 

#cmd = '"/Library/Application Support/ETVComskip/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

# MacPorts 64-bit wine

#cmd = '"/Applications/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

cmd = '"/opt/local/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

 

# nice the wine command

cmd = "/usr/bin/nice -n 14 " + cmd

 

EyeTV software

EyeTV uses the triggered AppleScripts RecordingStarted and RecordingDone in the directory /Library/Application\ Support/EyeTV/Scripts/TriggeredScripts. Open these using the AppleScript Editor and use these scripts instead. You will probably have to Save As... on your desktop, then copy the updated script back into the correct directory to avoid permissions issues.

 

RecordingStarted.scpt

 

on RecordingStarted(recordingID)

    delay 10

    -- comskip81 uses ffmpeg and does not support live tv; put this in RecordingDone.scpt

    --    set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --log " & recordingID & " &> /dev/null &"

    -- display dialog cmd

    -- set cmd to "env > /tmp/etv_test.log &"

    -- do shell script cmd

 

    --disable this if you do not want a logfile written

    write_to_file((short date string of (current date) & " " & time string of (current date)) & "Recording Started run for ID: " & recordingID & (ASCII character 13), (path to "logs" as string) & "EyeTV scripts.log", true)

end RecordingStarted

 

on write_to_file(this_data, target_file, append_data)

    --from http://www.apple.com/applescript/sbrt/sbrt-09.html

    try

        set the target_file to the target_file as string

        set the open_target_file to open for access file target_file with write permission

        if append_data is false then set eof of the open_target_file to 0

        write this_data to the open_target_file starting at eof

        close access the open_target_file

        return true

    on error

        try

            close access file target_file

        end try

        return false

    end try

end write_to_file

RecordingDone.scpt

 

-- Run the python MarkCommercials script for the given recording

-- this must be run with the RecordingStarted script

-- it will check if there were multiple PIDs for the recording and runs MarkCommercials for each pid

-- requires updated MarkCommercials which allows specifying the pid

-- by Ben Blake, September 2009

 

-- modified for latest Comskip, which cannot be run until after recording is finished; waits for Turbo.264 HD to stop running as well

-- S.T.Smith

 

global LogMsg

 

on RecordingDone(recordingID)

 

    set ProcessName to "Elgato H.264 Decoder"

    set DelayTime to 60

    set MaxDelays to 4 * 60 -- four hours

 

    -- delay until the process slows down or stops running

    repeat MaxDelays times

        delay DelayTime -- delay at least once to give it a chance to start

        set pcpu to CPUPercentage(ProcessName)

        if pcpu is equal to "" or pcpu < 2.0 then exit repeat -- break out of delay loop if it's not running

    end repeat

 

    delay 10

    -- comskip81 uses ffmpeg and does not support live tv; take this from RecordingStarted.scpt

    set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --log " & recordingID & " &> /dev/null &"

    -- display dialog cmd

    -- set cmd to "env > /tmp/etv_test.log &"

    do shell script cmd

 

    set LogMsg to ""

    CheckMultiplePIDs(recordingID)

 

    --disable this if you do not want a logfile written

    if (count of LogMsg) > 0 then

        write_to_file((short date string of (current date) & " " & time string of (current date)) & LogMsg & (ASCII character 13), (path to "logs" as string) & "EyeTV scripts.log", true)

    end if

end RecordingDone

 

-- testing code: this will not be called when triggered from EyeTV, but only when the script is run as a stand-alone script

on run

    tell application "EyeTV.app"

        set rec to «class Unqu» of item 1 of every «class cRec»

 

        my RecordingDone(rec)

    end tell

end run

 

-- compute the percentage CPU used by ProcessName

on CPUPercentage(ProcessName)

    set ProcessPS to do shell script ("ps -axwwc | grep '" & ProcessName & "' | grep -v grep || true")

    if ProcessPS is not equal to "" then

        set ProcessID to word 1 of ProcessPS

        set ProcessPS to do shell script ("ps -xwwco pid,ppid,%cpu -p " & ProcessID & " | tail -1")

        set ProcessCPU to word 3 of ProcessPS

        return ProcessCPU as number

    else

        return ""

    end if

end CPUPercentage

 

on CheckMultiplePIDs(recordingID)

    --check if there are multiple Video PIDs in the file

 

    tell application "EyeTV.app"

        set input_text to my read_from_file((path to "logs" as string) & "ETVComskip" & ":" & recordingID & "_comskip.log")

        if (count of (input_text as string)) > 0 then

            set logdata to every paragraph of input_text

            set logdata_lastrow to (item ((count of logdata) - 1) of logdata) as string

 

            if (items 1 thru 19 of logdata_lastrow) as string = "Video PID not found" then

                --multiple Video PIDs, rerun MarkCommercials until successful

 

                set recrdingIDInteger to recordingID as integer

                set rec to «class cRec» id recrdingIDInteger

                set LogMsg to "RecordingDone found multiple PIDs for recording ID: " & recordingID & ", Channel " & («class Chnm» of rec) & " - " & («class Titl» of rec)

 

                set PIDs to (items 44 thru ((count of logdata_lastrow) - 2) of logdata_lastrow) as string

                set delims to AppleScript's text item delimiters

                set AppleScript's text item delimiters to ", "

                set PID_List to {}

                set PID_List to every word of PIDs

                set AppleScript's text item delimiters to delims

 

                repeat with pid in PID_List

                    my launchComSkip(recordingID, pid)

                    repeat while (my mcIsRunning())

                        delay 5

                    end repeat

                end repeat

 

            end if

        end if

    end tell

end CheckMultiplePIDs

 

on read_from_file(target_file)

    --return the contents of the given file

    set fileRef to (open for access (target_file))

    set txt to (read fileRef for (get eof fileRef) as «class utf8»)

    close access fileRef

    return txt

end read_from_file

 

on write_to_file(this_data, target_file, append_data)

    --from http://www.apple.com/applescript/sbrt/sbrt-09.html

    try

        set the target_file to the target_file as string

        set the open_target_file to open for access file target_file with write permission

        if append_data is false then set eof of the open_target_file to 0

        write this_data to the open_target_file starting at eof

        close access the open_target_file

        return true

    on error

        try

            close access file target_file

        end try

        return false

    end try

end write_to_file

 

on launchComSkip(recID, pid)

    if pid = "" then

        set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --force --log " & recID & " &> /dev/null &"

    else

        set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --force --log " & recID & " --pid=" & pid & " &> /dev/null &"

    end if

 

    do shell script cmd

end launchComSkip

 

on mcIsRunning()

    set processPaths to do shell script "ps -xww | awk -F/ 'NF >2' | awk -F/ '{print $NF}' | awk -F '-' '{print $1}' "

    return (processPaths contains "MarkCommercials")

end mcIsRunning


Mac mini Server (Mid 2010), Mac OS X (10.7.4), Lion Server, EyeTV HD, Turbo.264 HD
  • Brandon Riffel Level 1 Level 1 (0 points)

    This is awesome!  I've been an EyeTV/aTV user for 3 years and lost the ability to edit out the commercials automagically.  I got it marking them again on my most recent install, but would have to the compress in EyeTV and export to Apple tv manually to get rid of them.

     

    How can I contact you more directly for help on this?  I am happy to make a donation to commskipper, and I'd make an in kind gift of some iTunes credit to you for assistance!

     

    Thanks!

  • essandess Level 1 Level 1 (5 points)

    I got it marking them again on my most recent install, but would have to the compress in EyeTV and export to Apple tv manually to get rid of them.

     

    Unfortunately, I don't believe this is possible yet with EyeTV products -- they have six different triggered AppleScripts, none of which allow running comskip before transcoding starts -- there's only ExportDone, but no ExportStarted, which would allow you to run comskip beforehand.

     

    This may be a (very thin) silver lining because if you mark commercials before transcoding, EyeTV/Turbo.264 reads comskip's .edl file (that has the start/stop times of all  commercials it found) and clips out all marked regions, including any false positives from comskip. FPs happen. I've run it this way in the past with lores iPhone exports (that required absolutely no transcoding at all) and it sure is nice and saves a good amount of disk space, but you have to go back to the original if you ever have a false positive at an important part.

     

    It would also be nice if iOS/OSX/Apple TV respected .edl files for video playpack, thus allowing iOS/Apple TV commercial skipping on the device if you also provide the .edl, but I don't believe that this is supported at all.

     

    The only options I see are to request that Elgato support an ExportStarted triggered script, or to request that Apple support .edl video playback on iOS/Apple TV.

  • wayveilleux Level 1 Level 1 (0 points)

    Hi Brandon,

     

    We face the same problem with automatic export to AppleTV. Have you found a work around ?

  • Niccum Level 1 Level 1 (0 points)

    Have you made this work in Mountain Lion?  I am having some difficulty.

     

    1. X11 apparently is now being run by XQuartz?? (at least that is what I was guided to install after some error messages).  Message said that ML no longer fully supports X11, and XQuartz is needed.

     

    2. In step 3, did you just install the mac version (ETVComskip-2.0.2-10.6.dmg), or did you copy the files from the donor's version (comskip81_048_donators.zip) which has the latest comskip.exe files.

     

    3. Which part of the RecordingStarted.scpt did you edit.  My existing RecordingStarted.scpt is quite long.  Is your script intended to be the entire script of a segment that needs to be changed from the original?

     

    4. Same question as 3 but for the RecordingDone.scpt.

  • essandess Level 1 Level 1 (5 points)

    Yes, it works great in ML. The three changes are: download Xcode from the App Store, download XQuartz, and upgrade all the outdated MacPorts, specifically wine-devel:

     

    $ sudo port selfupdate

    $ sudo port upgrade outdated

     

    I installed the latest googlecode ETVComskip, then made the changes described above, including swapping in the files from the donor's version of Comskip into the directory /Library/Application Support/ETVComskip/comskip.

     

    That's my entire RecordingDone script. I also modified the Python script /Library/Application Support/ETVComskip/MarkCommercials.app/Contents/Resources/MarkCommercials.py to nice +15 all the wine jobs, though this isn't necessary.

     

    In fact, you've caught me coding up some Python to add the .edl times as chapter times in the h.264 files that EyeTV exports for iPad streaming and iTunes, so you don't have to fiddle with the annoying and innaccurate iOS video time swiper. This will, I hope, provide Comskip information on iPads. I'll give this a week or two to finish and test, then I'll post that too. The relevant command is mp4chaps, available via:

     

    $ sudo port install mp4v2

  • essandess Level 1 Level 1 (5 points)

    (It was faster and easier to finish and this than I expected.)

     

    ETVComskip works great for EyeTV's original recordings, but the Comskip information isn't included with exports to iPad/iPhone/Apple TV files created by EyeTV or Turbo.264. Result: iTunes exports and wifi streams have ads. This modification addresses this issue by adding Comskip .edl information as chapters within the H.264 .m4v files exported by EyeTV and Turbo.264, providing a mobile Comskip solution: when the ads appear, skip to the next chapter. This is a lot easier and faster than using the (barely functional) time swiping capability to guess where the show starts. It also avoids the problem of false detections by not deleting any program material.

     

    See the script and details at code.google.com/p/etv-comskip/issues/detail?id=54.

  • essandess Level 1 Level 1 (5 points)

    The only issue I would highlight is that while iTunes Sharing over wifi supports navigation by by mp4 chapters, the iOS EyeTV app does not. Please contact Elgato and request this capability if you want to use their app for tv shows free of commercials on your mobile device. But iTunes just works.

  • Niccum Level 1 Level 1 (0 points)

    Great.  Thanks for sharing all of this information.  It will really help to simplify the whole process.

     

    Interestingly, if I read you last post correctly, my EyeTV setup does indeed already skip the marked commercials in the 1080p AppleTV export.  I just check to ensure the skip areas are correct and simply export by pressing the 1080p AppleTV Export button.  I have not had to compress prior to export.  I have not tried anything but the 1080p Export.

     

    Sorry, but I don't understand the "I also modified the Python script /Library/Application Support/ETVComskip/MarkCommercials.app/Contents/Resources/MarkCommercials.py to nice +15 all the wine jobs, though this isn't necessary."

     

    So, both of the above recording scripts are in their entirety.  RecordingStart.scpt and RecordingDone.scpt

     

    Again, thanks for the info.

  • essandess Level 1 Level 1 (5 points)

    A hilarious aside: this discussion group will not allow posts with the (rot13.com) phrase "pbzzrepvny-serr gi". Try it!

  • essandess Level 1 Level 1 (5 points)

    > my EyeTV setup does indeed already skip the marked commercials in the 1080p AppleTV export.

     

    It depends. I've observed that if EyeTV is transcoding a new file (takes a long time, hence the desirability of a Turbo.264 dongle), AND there's an existing .edl file from Comskip, the the transcoded respects the .edl edit decision list and deletes the (hopefully) commercials. Too bad if there are false detects. If, however, the export to iTunes is the same format at the EyeTV sharing format, then the .edl is ignored.

     

    The scripts above do trivial load balancing by making sure that transcoding and/or exports are done first, then Comskip is run. The .m4v files in EyeTV's archive will have Comskip chapter markings, but any exports to iTunes prior to running Comskip will not have these chapters. My personal solution is to turn off automatic iTunes exports and export using the same quality as EyeTV sharing files, which ensures a fast simple copy to iTunes with no new transcoding when I export manually.

     

    Now that I think of it, a simple mod to take care of previously exported iTunes files would be to write an ExportDone script that saves the full file name of the iTunes exported file within its originating .eyetv directory, then when Comskip is run in the Python script MarkCommercials.py, look for this file and update its chapter information. This would be useful for people who save both 720p files for iPad streaming and 1080p files for Apple TV. That's a lot of disk space and transcoding cpu cycles. I just use 720p for all streaming and exports, and save the 1080p for an HDMI cable.

  • wayveilleux Level 1 Level 1 (0 points)

    Essandess,

     

    With the python script you're writing, does that mean we will be able to integrate the .edl file when using the auto export to iPad after recording (the missing ExportStarted) so we will be able to view TV Show on Apple TV without commercial ? That would be great ! Thanks for your time. (BTW, I also have the Elgato 264 Turbo HD)

  • gardnern Level 1 Level 1 (0 points)

    I think this will take care of my issues with videos that are recording getting messed up while commercials are being blocked out in Mountain Lion (an issue I never had in Lion oddly).

     

    I've also taken to exporting for iPhone now as well.

     

    I wonder if, instead of calling the routine on RecordingDone and waiting for the encoder to finish, you could just call the script on ExportDone?  I guess it all depends on if you do a lot of other exporting.

     

    Calling it on ExportDone would also cause it to re-run (with --force) after compacting.  I haven't messed around to see if the chapter marks mess with what gets shown if they aren't re-calculated after compacting.

     

    Also, I fixed up the Growl notification (--growl) from within the python script of MarkCommercials if anyone else is interested in having this work.

  • essandess Level 1 Level 1 (5 points)

    A lot of people, including me, only export a subset of their shows to iTunes, so handling commercial skipping with ExportDone would ignore commercial skipping for a lot of shows. And to avoid sucking all CPU cycles into h.264 trancoding and commercial marking, the RecordingDone script above waits for exports to be finished, which has the same functionality as putting this code in ExportDone.

     

    I've been using commercial skipping on an iphone/ipad for a week now and it's great!—just hit the right video arrow and it works like a charm. I've been exporting EyeTV to itunes by hand, which is getting old already. I'd like to wrote an ExportDone script that remembers the exported show filenames in a text file within the source .eyetv directory, then adds the mp4 chapter info to those files when MarkCommercials.py is invoked. If anyone already has such a script, please post it.

     

    And it would be great if you could post your updated Growl notification code for Mountain Lion. I'll update the ETVComskip post with it.

  • Brandon Riffel Level 1 Level 1 (0 points)

    Ok, so I'm trying to get caught up.  Are you all saying that recording in EyeTV, marking commercials, compressing, and exporting to AppleTV/iTunes, is working well.  Working in Mountain Lion?

     

    Is having the Turbo 264 a requirement?

     

    I have a current generation mid-range Mini acting as my media server with 2 USB tuners.  Elgato support said that I would probably only see marginal speed improvements with T264, so I haven't invested yet.

     

    I'm current recording and marking commercials with EyeTv Comskip, but then after the recordings are done, I go in and manually compress.  Then go back and manually export to ATV.  The compressing of a 30 minute HD program (down to 20 minutes after editing) tales about 5 minutes.  Export to ATV about 30, assuming I'm not recording or compressing anything else at the time.

     

    I'd love to be able to automate any or all of this.  I do like being able to fine tune the commercial markers, I can usually shave off at least 3 more minutes or network commercials for other shows I don't care about, and cut off the endings and beginnings of the surrounding programs that sometimes are included.

     

    Can somebody give a very basic breakdown of exactly what is working and in what scenario?  As we are all Apple owners, we are ok with paying for form and functionality, we just want it all to work seemlessly!

     

    Thanks again to all of you for this hard work.

     

    Again, I'm offering a $25 iTunes credit to whomever wants to remote into my setup and get it working/give me a tutorial.

Previous 1 2 3 Next