Getting my ex's contacts in my phone?!

So I took my ex's iPhone 4S a few years ago and it then became my phone as he got a new phone (a Samsung) and then when it broke off, I got brought the new iPhone 6S (A few weeks after it coming out) and logged onto my Apple ID and then re-downloaded my apps and everything was good. At the beginning of the year I went on my messages and typed in the send a new message bar and went to type in a contact and I found my ex's contacts in there too. I went through the whole alphabet and typed up each letter and there were 40+ contacts in there. I did the same thing today and I noticed that there were more contacts in there that I hadn't seen before and were clearly not mine. I have still kept the same number but he hasn't. How would I still be getting his contacts in my phone if it's a new phone and he doesn't have the same number still? Would he be getting my contacts in his phone? Has anyone had the same or similar issue and how can I fix this? All suggestions welcome! I have tried to contact Apple Support by chat but each time it doesn't let me contact them.