Mail incorrectly filtering junk mail

Mail has been sending messages from certain contacts to Junk, even though they are in my contact list and have been starred as VIPs. This is happening even for responses to emails I have sent them. How can I fix this without making a rule for these specific senders?