No, no one can track your iPhone by putting their contact information in the Contact app. People can follow you in the Find app, but you have to approve it yourself. You can check if you follow someone and if someone follows you here in the Find app.
Open the Find app, tap the People button at the bottom. If you see a contact in the list who follows you, swipe from right to left and delete the person. The person will then not be able to see your location and must request to see it. You can just reject it if it happens.
You can read more about Find People here:
Find people and share your location with Find My - Apple Support