Can't remember password for "Erase All Content and Settings"

I purchased a phone and the person logged out of their iCloud, the imei was clean, and everything worked great, but he forgot the 4-digit code when I asked him to factory reset. Assumed it was no big deal because I could log into my iCloud just fine and my sim card worked.

I have restored it with a backup from my iPad and everything worked great. However, when I try to do a full factory reset it still won't work without his 4-digit code, even though it's restored to my iPad and is set to my iCloud... how can this be?

Is there anything that can be done to ensure the phone has a 'clean start' when I set it up on my new line or am I screwed? What happens if I don't fully reset and just use it with my backup and iCloud, will he ever be able to access any of my information?
TIA!!!