iTunes 11.3.1.2 addresses the issue of iTunes taking forever to go through Podcast subscriptions, even when you are not subscribed to them!
However, users (including myself) have reported that following that update, when switching to Podcasts, the message "iTunes has stopped working" appears. I took Apple's advice on that problem, which is to start iTunes in Safe Mode. For me, that fixed the problem, but for anyone who continues to have the problem, their advice continues...
Read this article; iTunes: Troubleshooting issues with third-party iTunes plug-ins where you will find specific mention of iTunes for Mac (by the way, you have posted in the iTunes for Windows forum) and what to do if the problem persists.