Why do my iMessages take a long time to deliver?

I currently have an iPhone 5 running on iOS 6.1.4. For the most part my iMessages/texts will send just fine, but once in a while (and more recently, particularly when I text one of my friends who also has an iPhone) it will take a very long time, between 20-30 minutes or longer, to send a simple text message through iMessage. The time stamp shows when time I hit the send button but it doesn't say "Delivered" underneath my text bubble until 20 minutes or so. Why does this happen when I'm texting this one iPhone in particular? I've tried restarting my phone as it was still delivering but that didn't work. It eventually sent, but it takes a long time.