There is NO requirement for you assign anything to a contact. The only contact you need to be concerned with is YOUR contact. If you don't want a poster associated with your contact, when you call someone, don't use a poster. As to your contacts, if they choose a poster for their contact, it will show on your phone. I think you're making this much more complex than it is or needs to be. To be clear, you can turn off your own contact poster. But you cannot turn off posters your contacts may have chosen for themselves.
Perhaps this will help you --> Add or edit your contact info and photo on iPhone - Apple Support