Did you setup as new or restore the backup you had before you restored in iTunes? Best fix is restore, setup as new, then sync iTunes to get your contacts back etc.
However if you restored firmware (via iTunes) then restored a backup you can try the "reset all settings" option Davidch outlined. That will sometimes solve the issue after you restore the corruption from a backup. Keeps your data intact as well
i recommend you try my process below with your iPod touch 4th generation. you mention upgrading to iOS 5. if you have not updated to 5.0.1, be sure to do that first. then follow my process step by step and in order.
1. Reset all settings (settings app-> general-> reset)
2 a. Go through initial setup steps (lang, wifi, siri, enable location, etc) and choose setup as new phone (don't worry your apps, data, contacts, mail will still be there). Do NOT restore from iCloud or iTunes (It can copy back corrupt settings)
2 b. If you do not get the complete new setup screen with language setup and setup as new phone or restore from iTune/iCloud backup, be sure to go back to #1 and reset all settings again (it should happen the second time)
3. Turn off system location services timezone and iAd (settings -> location services -> system services)
4. Fully discharge battery (until you get the spinning wheel and it shuts off)
5. Fully recharge battery (overnight if possible)
In my experience this improves the Standby battery drain issue significantly in most cases. It reduces drain from 2-4% or more per hr to 0.5% or less. It has worked for many, many users now. If it does not work after a few try's you may have a real battery or hardware issue and should contact Apple. Good Luck!