I have the same issue. No AirPlay from iTunes to AppleTV with the following:
PC: Windows 7 Ultimate 64Bit. iTunes: 11.05.05 Apple TV: v5.3
I have tried almost everything from iTunes reinstalls to firewall disabling, multiple reboots, etc. Nothing will get iTunes to Airplay to my Apple TV.
Airplay works from my iphone (iOS 6.1.2) to my AppleTV.
And if I turn on AirPlay on the Apple TV I can see the icon appear in iTunes, if I turn it off in ATV, it disappears from iTunes, so there is some communication/detection happening.
But if I select my ATV, there is a delay of 5 to 10 seconds and then a message box appears stating the connection can not be made because of a connection issue or it has timed out.
I even set the ATV "Onscreen Code" on. and when I select the ATV from iTunes it displays the 4 digit number to enter. I enter the number and after 5 seconds, it reports it times out... So Apple whats happening here?
Now the reverse direction is possible. Via homeSharing, I can see my iTunes libary from Apple TV under Computers (HomeSharing has to be on in both ATV and iTunes), and I through this I can select and play movies or music on the ATV. Not ideal, but possible.
I believe it started when iTunes was last updated to v11.05.05.
Just to prove it, I started up an older VMWare session on the PC. The VM runs XP and has iTunes 10.6.1.7. It bridges to the Host System (Win 7 Ultimate) network interface card. I was able to easily send music and a movie to the AppleTV, so Airplay works just fine from this version.
Clearly this is an iTunes 11.05.05 problem with Apple TV 5.3
PS Apple Admins, I hope you getting the message... many people posting about this. It is a problem with the release of iTunes, not a configuration issue.
I am willing to send you any diagnostics or log files so you can see what is happening, just let me know the tools/steps to follow and I'll send you log files. Note: I have wireshark, just tell me the filter strings to use.
I have tried the most obvious troubleshooting, so please don't suggest I change or add to the firewall (as it makes no difference even if it is off), reboot the computer, router, ATV (been there done this many times), or reinstall iTunes (yes done this 2x).
What is really interesting is that I also have "AirParrot" and this works just fine to put my PC screen on my ATV, but iTunes 11.05.05 can not send a movie or music....this just reinforces the problem is an iTunes issue.