The same thing happened to me with an Apple TV 2 - trying to update to firmware v5.1.1 on the thing failed [this has happened with previous updates too] but this time it seemed to hang so I eventually I unplugged it from the power and HDMI and connected it with a microUSB cable to my iMac 27 and iTunes 11 recognized it without delay. However, when doing a restore, iTunes 11 returned the error "Device cannot be found."
Moving ahead, I downloaded the v5..1.1 .ipsw file and did an option-restore in iTunes 11. No problems with the restore.
However, it failed to boot up / show up when reconnected to the TV. On one attempt of this sequence of events, it did work properly for about 20 minutes before crashing and I was back to where I was.
Tried to restore on my MBP Retina, still with iTunes 11, same error message, so did an option-restore with the downloaded v5.1.1 .ipsw file. Same result — would not boot up when reconnected to the TV.
I thought the thing had been frizzed when I interrupted what appeared to be a stalled firmware update while connected to the TV and wireless network, but the fix turned out to be precisely what xgentryx surmised — it's iTunes 11.
Booted up Windows 7 in Parallels 7 on the MBP Retina and did a flawless regular restore in iTunes 10; no error messages and the thing is now working perfectly.
I'm tempted to test the obvious problem with iTunes 11 with an Apple TV 3 but I've spent enough time on this already. I imagine we'll be seeing an iTunes 11.01 update soon.
Thanks for the suggestion and I hope my experience is helpful.
Cheers!