Will iPad charger work for iPhone?

I have a trip coming up and will be taking along the iPad 2 and the iPhone 4. I'd only like to have to bring one charger for both, which is better the better one to bring? I've seen that the iPhone charger will charge the iPad much slower, and the iPhone will only charge at the speed it knows to. So it looks like the iPad charger is the one to bring for the quickest of the two, just making sure that is not going to harm anything, thanks!