Hi mmunger!
I think I have a solution for you:
According to this: http://www.macobserver.com/tmo/article/apples_safari_6_rss_blunder
there is only the option to edit the the plist: com.apple.LaunchServices.plist in ~/Library/Preferences
because the mentioned program for setting default Applications RCDefaultApp: http://www.rubicode.com/Software/RCDefaultApp/ did not work reliably according to him.
I could not find the entry that sets the behaviour for opening RSS feeds in that mentioned plist, so I just tried RCDefaultApp under Mavericks and it worked just fine. In the "Internet"-tab I selected RSS and then my default Application "Feedy", that I have just bought off the app store because it looked nice. (And it works nicely, too.) Now I just hope RCDefaultApp does not mess up other things.
That mentioned article is from the 8th of August 2012, maybe there is better solution in the meantime? Maybe there is a modern alternative to the mentioned RCDefaultApp? I could not find one, but I have not looked very hard, yet.
By the way: In my case RSS feeds were opened in my Gmail Fluid App. Each RSS entry opened a tab there, which meant that 927 tabs were opened at the same time which completely blocked my computer. Everytime this happened I had to force quit the app and then I had to delete the ~/Library/Saved Application State/com.fluidapp.FluidApp.[Name].savedState folder so that the Fluid App would not restore all the 927 tabs when I opened it again.
