What I have experienced is when you keep using the iPhone regularly the contacts with whom you interact recently and regularly will only appear here.
If you do not interact with a contact then it will disappear automatically.
However you may....
- To turn off Siri Suggestions, go to Settings  > Siri & Search, then turn off any of the following:
- Suggestions in Search
- Suggestions in Lookup
- Suggestions on Lock Screen
For a specific app, tap the app, then turn off Show Siri Suggestions.
Your personal information—which is encrypted and remains private—stays up to date across all your devices where you’re signed in with the same Apple ID. As Siri learns about you on one device, your experience with Siri is improved on your other devices.
- Siri is designed to protect your information, and you can choose what you share. To learn more, go to Settings  > Siri & Search > About Search Suggestions & Privacy.