Remove duplicate email addresses in Mail.app

At work I have my work email address (Exchange) and my private iCloud email address setup in Mail.app.

I have merged the contacts I have in both accounts, but when I compose a new email and type in the email or name in the "To:" field, it shows duplicate emails. (And sometimes I can't browse the list with the "down arrow" key)

How do I get mail to only show me one email address per contact?

I would hate do disable my iCloud completely as it syncs my co-worker contacts when I update them in the Exchange account?

(The reason I have the Exchange is that I have a lot of business contacts, I don't want to merge with my private iCloud)