Basically its a problem with Mail leaving an open socket for each IMAP folder. Take a gMail account, it has up to 9 folders, other IMAP accounts would normally have 5 (Inbox, Sent, trash, Junk, Deleted).
So in my case I have 2 gMail accounts and 5 other IMAP accounts, thats 42 connects constantly polling your ISP and clogging up your connection.
The genius' at Apple have know about this for nearly 7 years now and of course no fix. They seem to prefer the usual arrogant "it just works" line, if we don;t admit to a problem, there is no problem. Just google "OSX mail IMAP".
Anyway, if you turn off "use IDLE command f the server supports it" and set Mail to check for mail every 5 minutes it tends to sort it. Its a pita but that Apple for you... Also, if you have "push" turned on your iPhone and are tethering for a connection, changing to "fetch" will make a HUGE difference to your connection speed, basically due to the same open socket problem.
In a nut shell, its just lazy coding, AGAIN.
Otherwise use Thunderbird or another mail app.