how do i back up mails?

If you are using IMAP to connect to your host server, simply sign into the account on the new iMac (Mail > Preferences > Accounts, or System Preferences > Mail, Contacts, Calanders), and all your mail/mail folders will be fetched from the server. There is no need to transfer anything.

See here:

Mac OS X Mail: How to transfer downloaded email ... - Support - Apple

Or, use time machine:

See:

http://pondini.org/TM/15.html

http://pondini.org/TM/28.html

Is there a reason you chose not to use the Migration Assistant? That's by far the easiest way.

If you have already set up your new Mac, it's a bit more of a pain, but you can still use it.