Hello Cricketkris,
This page can help out with providing information on VIP senders within the Mac's Mail app: Show emails from VIP senders in Mail on Mac - Apple Support
Are the contacts showing up in the app's sidebar or the Favorites bar? If so, and the star still isn't present to select/remove for them then as an alternative way to remove someone as a VIP (or make them a VIP) you can try the following:
- In the Mail app open an email that was sent to them or received from them.
- Hover your mouse over their name in the From/To field.
- Click on the downward pointing arrow next to their name that appears when hovering.
- Click on "Add to VIPs" or "Remove from VIPs", depending on whether they're currently a VIP or not.
Cheers.