Might you have a very large "Previous Contacts" list? Here's how to maintain it (from Mail Help):
Manage addresses in the Previous Recipients list
When you send messages, Mail stores your recipients’ email addresses in the Previous Recipients list. When you want to send new messages to those email addresses, just type a few letters in an address field, and Mail automatically fills in the rest.
- Choose Window > Previous Recipients.
- In the Previous Recipients window, do any of the following:
- To sort the list by name, email address, or the date the address was last used, click a column header.
- To search for a name in the list, enter the name in the search field.
- To view the card for a recipient who is already in Contacts (a card icon precedes the recipient’s name), double-click the name. Contacts opens and displays the card.
- To add one or more selected recipients to Contacts, click “Add to Contacts.”
- To remove one or more selected recipients from the list, click Remove From List.Removing the recipient is useful if you don’t intend to use the address again or if you previously mistyped an address and want Mail to stop suggesting it as a matching address.
To select addresses that are next to each other in the list, hold down the Shift key as you click the first and last addresses. To select addresses that are not next to each other in the list, hold down the Command key as you click each address.
You can remove a recipient from the Previous Recipients list while reading a message you received or writing a message. Click the arrow in a Smart Address and choose “Remove from Previous Recipients List” from the pop-up menu.
If you use iCloud Contacts, your Previous Recipients list is available on your other Mac computers that use iCloud Contacts and have OS X Mountain Lion installed.