Your computer has not been hacked, your e-mail account has. This is very common. No access to your address book is needed, since the hacker can simply harvest e-mail addresses from all the messages on the server. (Or, if this was an iCloud e-mail account, the hackers actually do have access to your contacts, since those are synced to the cloud.)
You need to change the password on your e-mail account immediately! Also, if your account has any kind of feature to allow people access without your password, be sure to check that feature for unauthorized modifications. (The classic example of this is GMail's delegation feature.)
Once you have done that, you should change the password for any of your other online accounts that use the same password. Keep in mind that the hackers can probably find out about these accounts by snooping through your e-mail, and if the password is the same, they've got instant access. Be sure not to use the same password for different accounts in the future, if you made this mistake.