Q: Contacts not fully synched

Hi all, hoping someone has some ideas.  I created an encrypted backup of my iPhone 4 to my computer and then upgraded to iPhone 6s Plus.  I then restored from my encrypted backup.  All seems to have synched fine except only a few contacts came over.  I never synched or setup access for my old iPhone 4 with the cloud so don't believe  anything there to access.  I did setup access to the cloud on the 6s Plus, but i did not set it up to share data with all devices.  Anyone know why not all the contacts synched and how I can get the remaining contacts on to my new phone?