So all of this advice is solid and needs to be done in order to get out of this loop/dead battery after upgrade problem. I basically left my phone for dead, bought the 5 and decided to try and resurrect it recently to use as iPad/iPod.
Once I got it out of the recover loop using Tiny Umbrella my iPhone would get to the step past picking a language during initial/new iPhone setup and then ask me to connect to iTunes. (This was ONLY if connected to wall power.) As soon as I would disconnect wall power to plug into iTunes it would die. I played with it for days! I ordered a new battery, replaced it, no good. You will never believe what finally worked.
1. Take the back off and disconnect the battery terminal (leave battery in)
2. Leave disconnected for 1 minute
3. Reconnect battery and then plug into cord attached to computer (LEAVE THE BACK OFF)
This should allow you to setup as new iPhone and all will work well. I was so happy, then I put the back of the phone back on and when I hit the power button to make it sleep, it would not come back on again...back into same loop. I realized that if the back was off all worked well. Then I realized that there is a little spring connector by the battery conneciton that has a contact point on the back of the phone. So, I put a piece of electrical tape on the back of the phone over the contact point, close the phone and BINGO. Works like new. I have no idea what the contact point and spring piece do and will let someone else weigh in on that issue. Hope this helps someone out there.