If you restored your phone to factory, and tested it without adding any additional content the phone and are still experiencing battery issues, then you need to make an appointment at the Genius Bar of the nearest Apple Store, as there could be hardware issues with your phone. A restore to factory wipes everything off the phone and restores it to how it came from the factory, with the exception of the iOS. There really isn't anything much to be running at that point, so you should not experience any battery issues.
You did not report what was using the most battery percentage, so I'm not sure you tried everything I recommended, but you can see what Apple has to say.