My iPhone gets very warm, is that normal?

My iPhone 4s gets very very warm, to the point where I can't hold it for a long time. It's worse when its charging but I've noticed it when I've not used it for an hour or so. I don't use a case and everything else seems to be working okay. Is it normal or should I take it too an Apple store?