Contacts not fully synched

Hi all, hoping someone has some ideas. I created an encrypted backup of my iPhone 4 to my computer and then upgraded to iPhone 6s Plus. I then restored from my encrypted backup. All seems to have synched fine except only a few contacts came over. I never synched or setup access for my old iPhone 4 with the cloud so don't believe anything there to access. I did setup access to the cloud on the 6s Plus, but i did not set it up to share data with all devices. Anyone know why not all the contacts synched and how I can get the remaining contacts on to my new phone?