I deleted the app (as described above) on a 4S, then updated my new 5S from the backup of my 4S. The podcast app remained off (I was given the options to add it back again when I set up my 5S, and of course I said "no"). May I ask if iOS 7 has a better Podcast app?
Ever since I deleted the Podcast app and started using iCatcher (a paid app), I've been very happy. It doesn't sync to iTunes, and takes a little bit of getting used to. But I think it is much much better than the Podcast app from iOS 6, with much more power, and not so buggy. I have no clue how the Podcast app is in iOS 7.
Message was edited by: Rob Mitchell1, as my workaround didn't work as I expected... may need to play with it a bit more.