Tangled ICloud Accounts

About 4 years ago I bought an old iPhone off a friend of mine. I believe I factory re-set it at the time, but ever since then every Iphone I have bought and logged in on I weirdly get the old owners email address auto populating, I see his Subway card points pop up in the wallet area and I've noticed just recently that I can see most of their passwords in the settings for various websites.

I have no idea what's going on, I've just bought a new phone and it's all transferred over again. The new phone has been factory reset and all I have done is sign into my Apple ID and all the same information from that old owner has come through again - it's like our ICloud/apple ID's are tangled. Does anyone know how to fix this as I'm worried the old owner can see my passwords as well.

Thanks.