essandess

Q: EyeTV with Commercial Skipping HD DVR Server

I use EyeTV HD and Turbo.264 HD to record and transcode h.264 shows for playback on a Mac, Apple TV, iPad, iPhone. I also use a self-modified version of etvcomskip to detect and mark commercials, which are then automatically skipped at playback. This makes tv viewing a lot more pleasurable and efficient, and is especially helpful controlling advertiser access to children. In case this is useful to others, I've outlined the steps here:

 

  1. Eric Kaashoek's website Comskip and download his HD-capable Comskip software. "Commercial" commercial-skipping companies (ReplayTV, TiVo, Dish, etc.) are quickly sued out of existence or into submission, even if they simply provide a basic "30-second skip" feature. Comskip is open source beerware that you can download, install, and use yourself, and you can't sue anyone for publishing source code. And rather than providing a simple 30-second skip feature, it detects, marks, and provides that capability for automatic commercial skipping.
  2. Comskip a Windows executable, so you'll need WINE (WINdows Emulator) on OS X to run it. First, download and install Xcode (free from the Apple App store), then download and install and install Macports, then in Terminal, run "sudo port selfupdate; sudo port install wine-devel". Finally, make sure that /Applications/Utilities/X11.app is added to your list of applications to run at login (WINE uses X11 to display Windows windows). Yes, this is windows emulation, but on a half-decent box you'll see that commercial skipping runs at around twice real-time for h.264 HD recordings.
  3. Download and install etvcomskip (and PyeTV if you want EyeTV to work with Front Row). The 2010 version of etvcomskip doesn't work with Comskip's latest HD capabilities, and the latest Comskip mpeg libraries do not allow live tv processing, so install these files to get everything working. Once you done this, you'll have a commercial-skipping HD DVR box on your Mac that can stream to Apple TV, iOS, or export to iTunes.

Comskip software

This lives in the directory /Library/Application Support/ETVcomskip.

 

Edit the file ./comskip/comskip.ini to use these HD-tuned settings, tailored for US cable:

 

detect_method=127;1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all
max_avg_brightness=21;maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
non_uniformity=50; Set to 0 to disable cutpoints based on uniform frames
max_volume=20; any frame with sound volume larger than this will not be regarded as black frame
logo_at_bottom=1; Set to 1 to search only for logo at the lower half of the video, do not combine with subtitle setting
intelligent_brightness=1 ; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA

volume_slip=150

output_dvrmstb=0; Set to 1 if you're running DVRMS-Toolbox
live_tv=0; set to 1 if you use parallelprocessing and need the output while recording
live_tv_retries=4; change to 16 when using live_tv in BTV, used for mpeg PS and TS

 

Edit the file ./MarkCommercials.app/Contents/Resources/MarkCommercials.py to use WINE to run comskip.exe nicely. Grep for these changes:

 

EyeTV.launch()

 

msg = 'Error: unable to communicate with %s\n' % options.app

 

#cmd = '"/Library/Application Support/ETVComskip/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

# MacPorts 64-bit wine

#cmd = '"/Applications/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

cmd = '"/opt/local/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile

 

# nice the wine command

cmd = "/usr/bin/nice -n 14 " + cmd

 

EyeTV software

EyeTV uses the triggered AppleScripts RecordingStarted and RecordingDone in the directory /Library/Application\ Support/EyeTV/Scripts/TriggeredScripts. Open these using the AppleScript Editor and use these scripts instead. You will probably have to Save As... on your desktop, then copy the updated script back into the correct directory to avoid permissions issues.

 

RecordingStarted.scpt

 

on RecordingStarted(recordingID)

    delay 10

    -- comskip81 uses ffmpeg and does not support live tv; put this in RecordingDone.scpt

    --    set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --log " & recordingID & " &> /dev/null &"

    -- display dialog cmd

    -- set cmd to "env > /tmp/etv_test.log &"

    -- do shell script cmd

 

    --disable this if you do not want a logfile written

    write_to_file((short date string of (current date) & " " & time string of (current date)) & "Recording Started run for ID: " & recordingID & (ASCII character 13), (path to "logs" as string) & "EyeTV scripts.log", true)

end RecordingStarted

 

on write_to_file(this_data, target_file, append_data)

    --from http://www.apple.com/applescript/sbrt/sbrt-09.html

    try

        set the target_file to the target_file as string

        set the open_target_file to open for access file target_file with write permission

        if append_data is false then set eof of the open_target_file to 0

        write this_data to the open_target_file starting at eof

        close access the open_target_file

        return true

    on error

        try

            close access file target_file

        end try

        return false

    end try

end write_to_file

RecordingDone.scpt

 

-- Run the python MarkCommercials script for the given recording

-- this must be run with the RecordingStarted script

-- it will check if there were multiple PIDs for the recording and runs MarkCommercials for each pid

-- requires updated MarkCommercials which allows specifying the pid

-- by Ben Blake, September 2009

 

-- modified for latest Comskip, which cannot be run until after recording is finished; waits for Turbo.264 HD to stop running as well

-- S.T.Smith

 

global LogMsg

 

on RecordingDone(recordingID)

 

    set ProcessName to "Elgato H.264 Decoder"

    set DelayTime to 60

    set MaxDelays to 4 * 60 -- four hours

 

    -- delay until the process slows down or stops running

    repeat MaxDelays times

        delay DelayTime -- delay at least once to give it a chance to start

        set pcpu to CPUPercentage(ProcessName)

        if pcpu is equal to "" or pcpu < 2.0 then exit repeat -- break out of delay loop if it's not running

    end repeat

 

    delay 10

    -- comskip81 uses ffmpeg and does not support live tv; take this from RecordingStarted.scpt

    set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --log " & recordingID & " &> /dev/null &"

    -- display dialog cmd

    -- set cmd to "env > /tmp/etv_test.log &"

    do shell script cmd

 

    set LogMsg to ""

    CheckMultiplePIDs(recordingID)

 

    --disable this if you do not want a logfile written

    if (count of LogMsg) > 0 then

        write_to_file((short date string of (current date) & " " & time string of (current date)) & LogMsg & (ASCII character 13), (path to "logs" as string) & "EyeTV scripts.log", true)

    end if

end RecordingDone

 

-- testing code: this will not be called when triggered from EyeTV, but only when the script is run as a stand-alone script

on run

    tell application "EyeTV.app"

        set rec to «class Unqu» of item 1 of every «class cRec»

 

        my RecordingDone(rec)

    end tell

end run

 

-- compute the percentage CPU used by ProcessName

on CPUPercentage(ProcessName)

    set ProcessPS to do shell script ("ps -axwwc | grep '" & ProcessName & "' | grep -v grep || true")

    if ProcessPS is not equal to "" then

        set ProcessID to word 1 of ProcessPS

        set ProcessPS to do shell script ("ps -xwwco pid,ppid,%cpu -p " & ProcessID & " | tail -1")

        set ProcessCPU to word 3 of ProcessPS

        return ProcessCPU as number

    else

        return ""

    end if

end CPUPercentage

 

on CheckMultiplePIDs(recordingID)

    --check if there are multiple Video PIDs in the file

 

    tell application "EyeTV.app"

        set input_text to my read_from_file((path to "logs" as string) & "ETVComskip" & ":" & recordingID & "_comskip.log")

        if (count of (input_text as string)) > 0 then

            set logdata to every paragraph of input_text

            set logdata_lastrow to (item ((count of logdata) - 1) of logdata) as string

 

            if (items 1 thru 19 of logdata_lastrow) as string = "Video PID not found" then

                --multiple Video PIDs, rerun MarkCommercials until successful

 

                set recrdingIDInteger to recordingID as integer

                set rec to «class cRec» id recrdingIDInteger

                set LogMsg to "RecordingDone found multiple PIDs for recording ID: " & recordingID & ", Channel " & («class Chnm» of rec) & " - " & («class Titl» of rec)

 

                set PIDs to (items 44 thru ((count of logdata_lastrow) - 2) of logdata_lastrow) as string

                set delims to AppleScript's text item delimiters

                set AppleScript's text item delimiters to ", "

                set PID_List to {}

                set PID_List to every word of PIDs

                set AppleScript's text item delimiters to delims

 

                repeat with pid in PID_List

                    my launchComSkip(recordingID, pid)

                    repeat while (my mcIsRunning())

                        delay 5

                    end repeat

                end repeat

 

            end if

        end if

    end tell

end CheckMultiplePIDs

 

on read_from_file(target_file)

    --return the contents of the given file

    set fileRef to (open for access (target_file))

    set txt to (read fileRef for (get eof fileRef) as «class utf8»)

    close access fileRef

    return txt

end read_from_file

 

on write_to_file(this_data, target_file, append_data)

    --from http://www.apple.com/applescript/sbrt/sbrt-09.html

    try

        set the target_file to the target_file as string

        set the open_target_file to open for access file target_file with write permission

        if append_data is false then set eof of the open_target_file to 0

        write this_data to the open_target_file starting at eof

        close access the open_target_file

        return true

    on error

        try

            close access file target_file

        end try

        return false

    end try

end write_to_file

 

on launchComSkip(recID, pid)

    if pid = "" then

        set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --force --log " & recID & " &> /dev/null &"

    else

        set cmd to "export DISPLAY=:0.0; /usr/bin/nice -n 5 '/Library/Application Support/ETVComskip/MarkCommercials.app/Contents/MacOS/MarkCommercials' --force --log " & recID & " --pid=" & pid & " &> /dev/null &"

    end if

 

    do shell script cmd

end launchComSkip

 

on mcIsRunning()

    set processPaths to do shell script "ps -xww | awk -F/ 'NF >2' | awk -F/ '{print $NF}' | awk -F '-' '{print $1}' "

    return (processPaths contains "MarkCommercials")

end mcIsRunning



<Links Edited By Host>

Mac mini Server (Mid 2010), Mac OS X (10.7.4), Lion Server, EyeTV HD, Turbo.264 HD

Posted on Dec 5, 2015 1:28 PM

Close

Q: EyeTV with Commercial Skipping HD DVR Server

  • All replies
  • Helpful answers

first Previous Page 10 of 10
  • by essandess,

    essandess essandess Dec 5, 2015 6:54 PM in response to the1greatgumbino
    Level 1 (28 points)
    Applications
    Dec 5, 2015 6:54 PM in response to the1greatgumbino

    Looks like the compiled binaries run fine. What happens when you just run

     

    /Library/Application\ Support/ETVComskip/bin/MarkCommercials

     

    without filtering the output through grep?

     

    Try recording a show, figure out its record id from the lhs of the MarkCommercials command, then run

     

    /Library/Application\ Support/ETVComskip/bin/MarkCommercials --log 0123456789 <insert correct record id>

     

    Does that work to mark commercials?

  • by the1greatgumbino,

    the1greatgumbino the1greatgumbino Dec 5, 2015 8:59 PM in response to the1greatgumbino
    Level 1 (0 points)
    Dec 5, 2015 8:59 PM in response to the1greatgumbino

    To clarify a couple questions essandess asked on the other thread that I was able to see via my email:

     

    Essandess - By the way, did I see that you're installing this under root's home directory? Do you run EyeTV as the same user?

    Yes.  I only have one User account, which is the Admin account.  When I run the installer, it automatically installs under the root directory

    /Library/Application\ Support/ETVComskip

    I do not have any EyeTV or ETVComskip files in the home directory:

    /Users/thegreatgumbino/Library/Application\ Support/

     

    Essandess - You're running 10.11, right?

    Yes, 10.11.1

     

    Essandess - Looks like the compiled binaries run fine. What happens when you just run

     

    /Library/Application\ Support/ETVComskip/bin/MarkCommercials

     

    without filtering the output through grep?

     

    Try recording a show, figure out its record id from the lhs of the MarkCommercials command, then run

     

    /Library/Application\ Support/ETVComskip/bin/MarkCommercials --log 0123456789 <insert correct record id>

     

    Does that work to mark commercials?

     

    The first string you list above was successful in pulling up the recording id's this time.  Ran the second string and it did indeed mark the commercials in the file upon playback in EyeTV. 

     

    MarkCommercials --log 471042878

            MarkCommercials    1.1

     

    1. Processing "SEC Football Championship" on [KHOU HDTV] channel [11-1]...

      Channel: 11-1 , not skipped

      Title: SEC Football Championship , not skipped

      Station name: KHOU HDTV , not skipped

  • by essandess,

    essandess essandess Dec 6, 2015 2:40 AM in response to the1greatgumbino
    Level 1 (28 points)
    Applications
    Dec 6, 2015 2:40 AM in response to the1greatgumbino

    That's all working correctly.

     

    Are you being patient for the triggered scripts to start and complete? It takes a few minutes for RecordingDone to launch MarkCommercials, then as you can see it takes ~10 minutes per hour of recording to mark the commercials.

     

    If that's not the issue, use a text editor to look at the EyeTV triggered script log here:

     

    ~/Library/Logs/EyeTV\ scripts.log

     

    What does that say for a specific show? You can delete that log file and run a test recording if it has numerous entries. Be sure to wait to see if the comskip process is triggered after the recording has completed.

  • by the1greatgumbino,

    the1greatgumbino the1greatgumbino Dec 6, 2015 8:36 AM in response to essandess
    Level 1 (0 points)
    Dec 6, 2015 8:36 AM in response to essandess

    essandess - Are you being patient for the triggered scripts to start and complete? It takes a few minutes for RecordingDone to launch MarkCommercials, then as you can see it takes ~10 minutes per hour of recording to mark the commercials.

     

    If that's not the issue, use a text editor to look at the EyeTV triggered script log here:

     

    ~/Library/Logs/EyeTV\ scripts.log

     

    What does that say for a specific show? You can delete that log file and run a test recording if it has numerous entries.

     

    Yes and no.  First, does playing the file in EyeTV prior to this time period terminate the script?  I had a couple that I did so yesterday that no commercials were marked.  However, there were also a couple that ran with no interference that did not mark commercials.  Regardless, the log shows RecordingStarted & RecordingDone was successful for all of them.

     

    12/5/15 4:58:10 PMRecordingStarted run for ID: 471049080

    12/5/15 6:58:11 PMRecordingStarted run for ID: 471056280

    12/5/15 7:02:00 PM RecordingDone run for ID: 471049080

    12/5/15 8:32:00 PM RecordingDone run for ID: 471056280

    12/5/15 9:58:10 PMRecordingStarted run for ID: 471067080

    12/5/15 11:32:00 PM RecordingDone run for ID: 471067080

    12/6/15 6:16:05 AMRecordingStarted run for ID: 471096954

    12/6/15 6:26:02 AM RecordingDone run for ID: 471096954

    12/6/15 6:29:55 AMRecordingStarted run for ID: 471097785

    12/6/15 6:30:01 AM RecordingDone run for ID: 471097785

    12/6/15 6:43:15 AMRecordingStarted run for ID: 471098585

    12/6/15 6:50:01 AM RecordingDone run for ID: 471098585

     

    essandess - Be sure to wait to see if the comskip process is triggered after the recording has completed.

     

    Looking at Activity Monitor, ComSkipper is always running regardless of whether or not there is a recording in progress.  I did not see any other comskip related activity start after recordings 471097785 or 471098585 finished recording.

     

    Variations

    Comskip also allows the capability for live commercial detection and skipping. This feature was once supported in the older Google code version of etv-comskip for non-HD recordings. etv-comskip has been refactored for sequential operations on HD recordings, but it would be possible to create a live commercial skipping.

    Does anyone have any guidance on how to accomplish live commercial detection and skipping with ETVComskip?  My old setup worked this way and I'd like to recreate this function as most of my TV viewing occurs after the show has started and before it has ended.  This was done purposely in order to use ETVComskip to breeze through commercials.

     

    <Link Edited By Host>

  • by the1greatgumbino,

    the1greatgumbino the1greatgumbino Dec 6, 2015 7:36 AM in response to the1greatgumbino
    Level 1 (0 points)
    Dec 6, 2015 7:36 AM in response to the1greatgumbino

    Forgot to mention that TriggeredScriptBridge does show up in Activity Monitor after the recording is done.

  • by essandess,

    essandess essandess Dec 6, 2015 8:47 AM in response to the1greatgumbino
    Level 1 (28 points)
    Applications
    Dec 6, 2015 8:47 AM in response to the1greatgumbino

    Please describe your EyeTV workflow, specifically whether you perform automated exports or not in

     

    Preferences>iPhone>More Options…>Prepare all new recordings for iPhone/iPad

     

    None of us has seen this before so it may be something to do with your specific configuration isn't triggering comskip.

     

    Re other questions:

     

    Playing a show will have no effect on commercial marking.

     

    ComSkipper is a helper app that's always running looking for commercials to skip through if they exist.

     

    The old live tv option was okay for ancient 480i content, but the CPU loads for 1080i h.264 content are too demanding to record/transcode/comskip all at once. And EyeTV will delete export content marked by comskip, which causes irreparable false alarm issues. That's why we export then mark commercials.

  • by essandess,

    essandess essandess Dec 6, 2015 9:27 AM in response to the1greatgumbino
    Level 1 (28 points)
    Applications
    Dec 6, 2015 9:27 AM in response to the1greatgumbino

    Another strong possibility is some permissions issue. It appears that you're running with both a user and root account. Create test user, non-admin account, install etv-comskip from this account, then record using EyeTV using this account. Does this fix the issue?

     

    By the way, running normal applications with the root account is asking for trouble.

  • by the1greatgumbino,

    the1greatgumbino the1greatgumbino Dec 7, 2015 6:27 PM in response to essandess
    Level 1 (0 points)
    Dec 7, 2015 6:27 PM in response to essandess

    Please describe your EyeTV workflow, specifically whether you perform automated exports or not

     

    I don't use automated exports.  My workflow is EyeTV to record & playback.  If I want to save the program, I manually tweak commercial markers & export from EyeTV.  I then transfer the file to my external server for Plex playback.

     

    Another strong possibility is some permissions issue. It appears that you're running with both a user and root account. Create test user, non-admin account, install etv-comskip from this account, then record using EyeTV using this account. Does this fix the issue?

     

    By the way, running normal applications with the root account is asking for trouble.

     

    Alright, got it working now.  I was perfectly content running a single root account, though ignorant to it's negative aspects.  Thanks for pointing that out.  After reading up on it, I did a clean install of El Capitan and setup a proper user account.  Loaded ETVComskip-3.4.1-10.11.dmg, updated files with comskip81_089_donators and it's working now.  I would point out that when I installed ETVComskip when logged in as user, it installed the files in the root/Library/Application Support/ETVComskip directory just as before when I installed it logged in as root.

     

    Thanks for all the help!!!  I can't thank you enough for your hard work in getting this operational.

     

    On another note, can anyone point me towards the most current / best information for tweaking the Comskip settings when commercial markers aren't quite lining up?

     

    <This link is a direct download>

  • by essandess,

    essandess essandess Dec 7, 2015 3:13 PM in response to the1greatgumbino
    Level 1 (28 points)
    Applications
    Dec 7, 2015 3:13 PM in response to the1greatgumbino

    Glad that it works.

     

    TThe settings in there are good for many U.S. cable shows, but definitely not perfect. Thing is, you could perfect for one set of shoes, then be off for others. If you do find comdkip.ini settings you like, please post them.

     

    Erik actively maintains and improves comskip, so please request that he take a the comskip forum, http://www.kaashoek.com/comskip/index.php.

    Also take note that this is Comskip donator code we're running, so please consider donating to Comskip, especially while requesting support.

  • by tomar012,

    tomar012 tomar012 Feb 19, 2016 7:07 PM in response to essandess
    Level 1 (8 points)
    Mac OS X
    Feb 19, 2016 7:07 PM in response to essandess

    Can someone tell me the how to make chapters out of the edl file produced by EyeTV?  I have mp4v2 but I'm having trouble getting the mp4chaps command to work for me. I keep getting the following error: ERROR: unable to open for write:

     

    I am grabbing (not exporting) the mpg file from the EyeTV Archive folder and transcoding it to m4v files.

     

    Thanks

first Previous Page 10 of 10