EyeTV with Commercial Skipping HD DVR Server

I have an HDHomeRun which outputs MPEG-2. The process that runs on my machine from EyeTV is "Elgato MPEG-2 Decoder" so I had to tweak the script to check that process for pausing to wait for the export to finish.

Below is the MarkCommercials.py with Growl fixed up. I know it works with Growl 2.0 but I think it should work back to 1.3. The script has to be called with --growl on the command line.

I also added a --m4vonly that will just chapter the m4v files on stuff that has been previously marked.

This is the first time I've coded in python, it all works but it might be a tad crude.

#!/usr/bin/env python # # MarkCommercials.py, EyeTV3 version # # Copyright (c) 2008, Jon A. Christopher # Copyright (c) 2008, TeamSTARS Dick Gordon and Rick Kier # Licensed under the GNU General Public License, version 2 # # # Run comskip on the specified file and replace the markers in the .eyetvr # file with the new markers. # # if no arguments show id, recording name... # # If argument is 'all', process all recordings which don't # have markers and do not match any exclude information from the cfg file. # # If argument is'forceall', process all recordings except those that match # any exclude information in the cfg file. # # Otherwise, argument is treated as an EyeTV recording id, and that # recording is processed if it is not excluded and it does NOT have any markers. # # To be done: # exporting - not needed?? # exclude channels # Issue: some channels are 0 # exclude titles # exclude station names # Issue: some station names are blank # Catch ^c when in c program - comskip throws SIGINT's away?? (see mpeg2dec.c) # Catch ^c when in python program - coded # EXCLUDED_TITLES and EXCLUDED_CHANNELS are NOT empty lists # Handle multiple video pids. # Added argument for PID - Ben Blake September 2009 # STS # Added macports wine, nice-ness, mp4chaps chapters headings at commercials import sys, os, string, os.path import time import math import traceback from optparse import OptionParser from ConfigParser import SafeConfigParser from os import listdir from os.path import isfile, join # Exit Codes # Everything worked ok successExitCode = 0 # Unable to import appscript importExitCode = 1 # Unable to find the recoring specified noRecordingExitCode = 2 # Error getting recordings from EyeTV getRecordingsErrorExitCode = 3 # Unknown exit code from comskip unknownComskipErrorExitCode = 4 # Error when accessing plist file accessPlistFileErrorExitCode = 5 # Error when accessing config file accessConfigFileErrorExitCode = 6 # Keyboard interrupt (^c) keyboardInterruptExitCode = 7 # Unable to communicate with the application communicationsErrorExitCode = 8 # provided with appscript package try: import aem from appscript import * except ImportError, e: sys.stderr.write('Error: importing appscript\n%s\n' % e) sys.exit(importExitCode) version = '0.4.0' # Cfg file definitions and variables userSectionName = 'User Section' listDelimiterName = 'LIST_DELIMITER' excludedTitlesName = 'EXCLUDED_TITLES' excludedChannelsName = 'EXCLUDED_CHANNELS' excludedStationNamesName = 'EXCLUDED_STATION_NAMES' excludedTitles = [] excludedChannels = [] excludedStationNames = [] # General variables options = None args = None recordingCount = 0 comskipLogPathName = None log = None growl = None eyeTV = None pathToComskip = None nameOfComskip = 'comskip' comskipLocations = ['.', r'/Library/Application Support/ETVComskip'] # for debugging. when False, will not actually run comskip, but will do everything else RUN_COMSKIP = True # Get the executable directory ETVComskipDir = os.path.abspath(os.path.dirname(__file__)) # Growl support commercialStart = 'Start' commercialStartDescription = 'Start detecting commercials' commercialStop = 'Stop' commercialStopDescription = 'Stop detecting commercials' programName = 'Mark Commercials' allNotificationsList = [commercialStart, commercialStop] def InitGrowl(): """docstring for InitGrowl""" global growl global allNotificationsList # Should we use growl? if not options.growl: # No return growl = app('Growl') enabledNotificationsList = allNotificationsList WriteToLog('Registering with growl\n') try: growl.register(as_application=programName, all_notifications=allNotificationsList, default_notifications=enabledNotificationsList) except Exception, e: WriteToLog('Error: registering with growl\n %s\n' % e) growl = None def sendGrowlNotification(name, title, description): """docstring for sendGrowlNotification""" # Send a Notification... if growl is not None: WriteToLog('Sending notification via growl (%s, %s, %s)\n' % (programName, title,description)) try: growl.notify(with_name=name, title=programName, description=description, application_name=programName) except Exception, e: WriteToLog('Error: growl notify\n %s\n' % e) # Create the log file def GetLog(name=None): """docstring for GetLog""" global log global comskipLogPathName # Should we log? if not options.log: return # Is the log directory created? fullPath = os.path.expanduser('~/Library/Logs/ETVComskip') if not os.path.isdir(fullPath): # No, create it. os.mkdir(fullPath) # Create the log if name is None: name = time.strftime('%m-%d-%Y %H-%M-%S') comskipLogPathName = os.path.join(fullPath, name + '_comskip.log') log = open(os.path.join(fullPath, name + '.log'), 'w') def WriteToLog(message): """docstring for WriteToLog""" if options.log: if type(message) == type(u""): message=message.encode("utf-8") log.write('%s - %s' % (time.asctime(), message)) log.flush() def CheckForApplicationCommunications(retries=3): """docstring for CheckForApplicationCommunications""" global EyeTV # launch the application EyeTV.launch() WriteToLog('Checking communications to %s with %d retries\n' % (options.app, retries)) for attempt in range(retries): try: # Get the recordings EyeTV.recordings.get() # It worked - break out of here WriteToLog(' Attempt %d worked\n' % (attempt + 1)) break except Exception, e: WriteToLog(' Attempt %d failed\n %s\n' % ((attempt + 1), e)) EyeTV = app(options.app) time.sleep(0.5) continue else: msg = 'Error: unable to communicate with %s\n' % options.app WriteToLog(msg) sys.stderr.write(msg) sys.exit(communicationsErrorExitCode) def GetRecordings(retries=0): """docstring for GetRecordings""" global EyeTV WriteToLog('Getting recordings\n') try: recordings = EyeTV.recordings.get() except Exception, e: msg = 'Error: unable to get recordings\n%s\n' % e WriteToLog(msg) sys.stderr.write(msg) sys.exit(getRecordingsErrorExitCode) WriteToLog('Recordings: %s\n' % recordings) return recordings # Possibly run comskip and return the name of a plist file with commercial markers in it def GetPlistFile(etvr_file, run_comskip=True): FileDir = os.path.dirname(etvr_file) dir, fil = os.path.split(etvr_file) FileRoot, ext = os.path.splitext(fil) MpgFile = FileRoot + ".mpg" PlistFile = FileRoot + ".edl" #cmd = '"/Library/Application Support/ETVComskip/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile # MacPorts 64-bit wine #cmd = '"/Applications/Wine.app/Contents/Resources/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile cmd = '"/opt/local/bin/wine" "/Library/Application Support/ETVComskip/comskip/comskip.exe" --ini="/Library/Application Support/ETVComskip/comskip/comskip.ini" "%s"' % MpgFile if options.pid <> "": cmd += " --pid=" + options.pid outputName = '/dev/null' if options.log: cmd += ' > %s 2>&1' % comskipLogPathName else: cmd += ' > %s 2>&1' % '/dev/null' if options.verbose: cmd += ' --verbose=%d' % options.verbose # nice the wine command cmd = "/usr/bin/nice -n 14 " + cmd WriteToLog('Changing directory to %s\n' % FileDir) os.chdir(FileDir) if run_comskip: #get the show name from the Directory Path showName=os.path.splitext(os.path.basename(FileDir))[0] # Notify the user sendGrowlNotification(commercialStart, commercialStart + " " + showName, commercialStartDescription + " on " + showName) # TBD stop comskip when ^c happens if not options.m4vonly: WriteToLog('Running: %s\n' % cmd) rc = os.system(cmd) else: WriteToLog('Skipped Comm Search, attempting to just add m4v Markers.') rc=0 # Add the Comskip information as chapters to all m4v files mp4chaps_all_m4v(FileDir) # Notify the user sendGrowlNotification(commercialStop, commercialStop + " " + showName, commercialStopDescription + " on " + showName) WriteToLog('Return code is: %d, 0x%x\n' % (rc, rc)) errorCode = (rc >> 8) & 0xff WriteToLog('Error code is: %d, 0x%x\n' % (errorCode, errorCode)) # Error code: # 3 = no Video pid found # 2 = Can't open the mpeg2 file TBD # 1 = Commercials found # 0 = Commercials not found if errorCode == 2: msg = 'Unable to open mpeg2 file return from "comskip": %d\n' % errorCode WriteToLog(msg) sys.stderr.write(msg) return None elif errorCode == 3: msg = 'No video pid found return from "comskip": %d\n' % errorCode WriteToLog(msg) sys.stderr.write(msg) return None elif errorCode == 1: WriteToLog('Commercials found by comskip\n') return PlistFile elif errorCode == 0: WriteToLog('No commercials found by comskip\n') return None else: msg = 'Error: unknown error code from comskip: %d, assuming it worked.\n' % errorCode WriteToLog(msg) return PlistFile # return start and ending times for the given line def TimeChop(line): fields = line.split("\t") start = (float(fields[0])) end = (float(fields[1])) return (start,end) # given a plist file, return a markers array suitable for adding to a recording def GetMarkersArray(PlistFile): try: file = open(PlistFile) lines = file.readlines() file.close() except Exception, e: msg = 'Error: accessing %s\n%s\n' % (PlistFile, e) WriteToLog(msg) sys.stderr.write(msg) sys.exit(accessPlistFileErrorExitCode) WriteToLog('Plist file contents: %s\n' % lines) markers=[] for line in lines: WriteToLog("Processing plist file line: '%s'\n" % line) start,end = TimeChop(line) WriteToLog('Adding marker, start: %d, end: %d\n' % (start, end)) marker = {} marker['position'] = start marker[aem.AEType('leng')] = end - start markers.append(marker) return markers def ProcessRecording(recording, run_comskip): global recordingCount channel = recording.channel_number() title = recording.title() stationName = recording.station_name() recordingCount += 1 msg = '%2d. Processing "%s" on [%s] channel [%s]...' % (recordingCount, title, stationName, channel) WriteToLog('%s\n' % msg) print msg.encode("utf-8") # Should excludes be allowed? if not options.noexclude: # Yes, Is this one allowed? # User can exclude titles, channels and station names # Channel msg=' Channel: %s' % channel print msg.encode("utf-8"), if str(channel) in excludedChannels: WriteToLog('Skipped due to channel match\n') print ' skipped' return print ', not skipped' # Title msg=' Title: %s' % title print msg.encode("utf-8"), if title in excludedTitles: WriteToLog('Skipped due to title match\n') print ' skipped' return print ', not skipped' # Station name msg=' Station name: %s' % stationName print msg.encode("utf-8"), if stationName in excludedStationNames: WriteToLog('Skipped due to station name match\n') print ' skipped' return print ', not skipped' # Get its path etvr_path = recording.location.get().path WriteToLog('Path to recording is %s\n' % etvr_path) # get the plist file for this recording, and make a markers array for it Plist = GetPlistFile(etvr_path, run_comskip) # Did we get a plist file? if Plist is not None: # Yes, convert it. markers = GetMarkersArray(Plist) # and finally, set them WriteToLog('Setting markers on recording\n') markers_string = str(markers) WriteToLog('Adding marker: %s\n' % (markers_string)) recording.markers.set(markers) mp4chaps = '/opt/local/bin/mp4chaps' def sec2hhmmss(secs): """Convert seconds to string HH:MM:SS.SSS format.""" rem = secs/3600 hh = int(rem) rem = (rem-hh)*60 mm = int(rem) rem = (rem-mm)*60 ss = int(rem) rem = (rem-ss) rem = '%.3f' % rem # millisecond precision rem = rem.replace('0.','.') return '%02d:%02d:%02d%s' % (hh,mm,ss,rem) def split_whitespace_nolibs(str): """Split a string by whitespace without using re or shlex libraries..""" strs = filter(None,str.split('\t')) strs = map(lambda s: s.split(' '),strs) strs = filter(None,[item for sublist in strs for item in sublist]) return strs def edl2mp4chaps(edl_file,file): """Convert an edl file into an mp4chaps file.""" txt_file = file.replace('.m4v','.chapters.txt') ftxt = open(txt_file,'w') comskipno = 0 comskipchapno = 0 lines = [line.strip() for line in open(edl_file)] for line in lines: times = map(float,split_whitespace_nolibs(line)) if (comskipno == 0 and times[0] != 0.0): comskipno += 1 if (len(times) < 2 or times[2] == 0.0): if (times[0] != 0.0): ftxt.write('%s Chapter %d End\n' % (sec2hhmmss(times[0]),comskipno)) else: ftxt.write('00:00:00.000 Beginning\n') comskipno += 1 ftxt.write('%s Chapter %d Start\n' % (sec2hhmmss(times[1]),comskipno)) else: # never seen this case, but here for logical consistency comskipchapno += 1 if (times[0] != 0.0): ftxt.write('%s Chapter %d Start\n' % (sec2hhmmss(times[0]),comskipchapno)) else: ftxt.write('00:00:00.000 Beginning\n') ftxt.write('%s Chapter %d End\n' % (sec2hhmmss(times[1]),comskipchapno)) ftxt.close() return def mp4chaps_all_m4v(dir): """Apply an edl file's entries to all m4v files in a directory.""" os.chdir(dir) edl_file = "" files = [ file for file in listdir(dir) if isfile(join(dir,file)) ] for file in files: if file.find('.edl') != -1: edl_file = file break if edl_file != "" and os.path.isfile(mp4chaps): for file in files: if file.find('.m4v') != -1: # remove all chapters cmd = mp4chaps + ' -r ' + file + ' > /dev/null 2>&1' rc = os.system(cmd) # create chapter file edl2mp4chaps(edl_file,file) # import chapters cmd = mp4chaps + ' -i ' + file + ' > /dev/null 2>&1' rc = os.system(cmd) return def main(): global options global args global excludedTitles global excludedChannels global excludedStationNames global log global EyeTV # Do the options usage = "usage: %prog [options] [RECORDING-ID | 'all' | 'forceall']" parser = OptionParser(usage=usage, version=version) parser.add_option("--noexclude", action="store_true", dest="noexclude", default=False, help="Do NOT exclude recordings specified in cfg file, default=%default") parser.add_option("--force", action="store_true", dest="force", default=False, help="Force commercial marking on specified RECORDING-ID. Allows marking when markers already exist, default=%default") parser.add_option("--growl", action="store_true", dest="growl", default=False, help="Enable growl notification, default=%default") parser.add_option("--log", action="store_true", dest="log", default=False, help="Enable logging, default=%default") parser.add_option("--app", dest="app", default='EyeTV', help="Specify EyeTV application name, default=%default") parser.add_option("--verbose", type='int', dest="verbose", default=0, help="Verbosity level, 0-10, default=%default") parser.add_option("--pid", dest="pid", default='', help="Specify the Video PID, default=%default") parser.add_option("--m4vonly", dest="m4vonly", default=False, help="Only chapter m4v files that have already had commercials marked, don't try to mark commercials. default=%default") (options, args) = parser.parse_args() if len(args): name = args[0] else: name = None GetLog(name=name) WriteToLog('%s, %s starting\n' % (sys.argv[0], version)) WriteToLog('Command line: %s\n' % sys.argv) WriteToLog('Application name: %s\n' % options.app) print '\t\t%s\t%s\n' % (os.path.splitext(os.path.basename(sys.argv[0]))[0], version) # Get the app EyeTV = app(options.app) # Use growl notifications InitGrowl() # Get our configuration file & data configInput = SafeConfigParser() try: cfgFilesRead = configInput.read([os.path.join(os.path.dirname(sys.argv[0]), 'MarkCommercials.cfg'), os.path.expanduser('~/Library/Application Support/ETVComskip/MarkCommercials.cfg')]) except Exception, e: msg = 'Error: reading configuration file\n%s\n' % e WriteToLog(msg) sys.stderr.write(msg) sys.exit(accessConfigFileErrorExitCode) WriteToLog('Cfg files read: %s\n' % cfgFilesRead) if cfgFilesRead: # Process the user's section if configInput.has_section(userSectionName): # Process the user's section # Get the delimiter listDelimiter = configInput.get(userSectionName, listDelimiterName) if configInput.has_option(userSectionName, excludedChannelsName): for channel in configInput.get(userSectionName, excludedChannelsName).split(listDelimiter): excludedChannels.append(channel) if configInput.has_option(userSectionName, excludedTitlesName): for title in configInput.get(userSectionName, excludedTitlesName).split(listDelimiter): excludedTitles.append(title) if configInput.has_option(userSectionName, excludedStationNamesName): for title in configInput.get(userSectionName, excludedStationNamesName).split(listDelimiter): excludedStationNames.append(title) WriteToLog('List Delimiter: %s\n' % listDelimiter) WriteToLog('Excluded Channels: %s\n' % excludedChannels) WriteToLog('Excluded Titles: %s\n' % excludedTitles) WriteToLog('Excluded Station names: %s\n' % excludedStationNames) # Test communications with application CheckForApplicationCommunications() # Get the location of the commercial skipper #findComskip() # Show the IDs and program names when there are no arguments # replace any non ascii characters with ? if len(args) == 0: for rec in GetRecordings(): programName = os.path.split(os.path.splitext(os.path.dirname(rec.location.get().path))[0])[1] outputName = '' for char in programName: # Insure the character is ascii if ord(char) <= 127: outputName += char else: outputName += '?' msg = ' %d = [%s], [%s], [%s]' % (rec.unique_ID.get(), outputName, rec.channel_number(), rec.station_name()) WriteToLog('%s\n' % msg) print msg.encode("utf-8") return successExitCode if args[0] == "all" or args[0] == "forceall": # batch mode, process all recordings without markers recs = GetRecordings() for rec in recs: markerCount = len(rec.markers.get()) WriteToLog('Marker count: %d\n' % markerCount) if markerCount == 0 or args[0] == "forceall": ProcessRecording(rec, RUN_COMSKIP) else: # triggered mode, process just the listed recording recs = GetRecordings() recordingRequested = int(args[0]) for rec in recs: if rec.unique_ID() == recordingRequested: WriteToLog('Found recording %d\n' % recordingRequested) break else: msg = 'Error: unable to find recording %d\n' % recordingRequested WriteToLog(msg) sys.stderr.write(msg) sys.exit(noRecordingExitCode) markerCount = len(rec.markers.get()) WriteToLog('Marker count: %d\n' % markerCount) # Recording already have markers? if markerCount == 0 or options.m4vonly: # No try: ProcessRecording(rec, RUN_COMSKIP) except Exception,e: exc_type, exc_value, exc_traceback = sys.exc_info() WriteToLog(repr(traceback.format_exception(exc_type, exc_value,exc_traceback))) # Is the recording already marked but the user wants it done again? elif markerCount != 0 and options.force: # Yes WriteToLog('Recording already marked - use forcing with --force option\n') ProcessRecording(rec, RUN_COMSKIP) else: # Recording already maked and user doesn't want it done again msg = 'Recording previously marked' WriteToLog('%s\n' % msg) print ' ', print msg.encode("utf-8") return successExitCode if __name__ == '__main__': ''' Call main ''' try: exitStatus = main() except KeyboardInterrupt, e: msg = 'Error: %s\n' % e WriteToLog(msg) sys.stderr.write(msg) sys.exit(keyboardInterruptExitCode) pass WriteToLog('Exiting\n') sys.exit(exitStatus)

I have been super excited to attempt to get this working on my install of EyeTV. The issue that I'm running into is that when I attempt to either run "MarkCommercials all" from the command line the result I get is this:

1. Processing "Castle" on [KSTP HDTV] channel [431]...

Channel: 431 , not skipped

Title: Castle , not skipped

Station name: KSTP HDTV , not skipped

Error: accessing 00000000162cf6e4.edl

(2, 'No such file or directory')

However when I run comskip directly from the command line using the following command (running from inside the recording's package):

/opt/local/bin/wine /Library/Application\ Support/ETVComskip/comskip/comskip.exe --ini=/Library/Application\ Support/ETVComskip/comskip/comskip.ini 00000000162cda88.mpg

There is a text file that is output named "00000000162cda88.txt". I've also been having issues with the program crashing but I'm betting that was one of my recordings being funky. Any thoughts on how to go about getting this to work? I'm getting the same result off of the RecordingDone.scpt as well. Any help would be appreciated. Thanks!

I'm not much for exporting to iTunes however pushing an appropriately named file to Plex would be pretty spiffy.

Does your export process do any renameing and/or file copying to specific locations?

Actually after I asked I went on a bit of a research trip. Right now I'm using a Scanner written to add links to the playable file in an EyeTV package into Plex. This works very well and even grabs weird stuff I record that dosn't have Episode numbers on them, like PBS cooking episodes.

Primarily I'm interested in the commercial skipping working within Plex itself. Whil this doesn't appear to be an option playing the recorded .mpg file directly, perhaps it would work if I exported into the eyeTV recording package and then had Plex play the .m4v file instead.

The question is if Plex will respect the chapter marks. If it won't then I'd have to run Comskip first, then have EyeTV run an export which would leave out the flagged commercial sections.

I have glanced over a lot of discussions on the Plex forums concerning its use of the improper (lower quality) stream. I believe a lot of work has been done in this respect. I know there is now a place to define your preferred stream quality at least for web based myPlex videos. In their new web client you can also define the quality of stream you wish to receive.

If you haven't looked into Plex within the last 6 months I would encourage you to give the latest release a try. If I could get football to live broadcast through Plex with Pause/FF/Rewind working I probably wouldn't watch anything at all through EyeTV directly..

I installed your latest files from the code.google site and works like a charm now. Thanks for all your help and hard work!

HI

thank you. your instructions were very helpful for me but I live in Europe and I don't have neither HD or turbo264. Are your changes to the standard version (edit markcommercials.py, the new scripts) useful for me? my etv comskip (standard version) marks well short program (below 1 and half hour) but not the longer ones (like movies).

if your changes are useful, could you explain better how to "edit markcommercials.py"? do I have to substitute the similar parts in the original file?

thank you

Hello,

Perhaps you can help me - I'm having a problem that initially was similar to sheldoa, i.e., that I was getting "No such file or directory" when running MarkCommercials from the command line. It doesn't appear to be the same problem, however, as when I look in my .eyetv file, I see no .edl file whatsoever. I came to the conclusion (correct me if I'm wrong here) that comskip didn't do it's job. So, I also tried running comskip from the command line:

/opt/local/bin/wine /Library/Application\ Support/ETVComskip/comskip/comskip.exe --ini=/Library/Application\ Support/ETVComskip/comskip/comskip.ini 0000000016536aa4.mpg

This fails with the error:

err:module:attach_process_dlls "winspool.drv" failed to initialize, aborting

err:module:LdrInitializeThunk Main exe initialization for L"Z:\\Library\\Application Support\\ETVComskip\\comskip\\comskip.exe" failed, status c0000005

Which leads me to believe I was right. It also seems like comskip is not functioning because wine itself is not starting. Does comskip itself create the .edl file within each .eyetv package?

I have carefully gone through the installation steps detailed in the readme, and have made sure I have the latest macports updates, the donor version of comskip, etc. XQuartz is installed and running, and wine-devel and mp4v2 and their dependencies install without error from macports. I also have copied your files from the zip as instructed. Is there something I'm missing?

Any help would be very much appreciated!

Cheers,

S

PS. This is Mountain Lion on a Mac Mini with my EyeTV Archive in a non-default location (on my external firewire Media volume)