Restoring Contacts without using iCloud

I need to restore my contacts. I do not keep contacts in ICloud because I have way too sensitive information there and I do not want it to be so accessible. So, for some reason my Contact information disappeared. Not all of it but most of it. When I research how to solve this problem it sent me to ICloud. Then I tried to use Time Machine, but it did not let me restore for some weird reason. So what do I do?

I think I had an issue like this a couple of years back and they told me to trash one file and find some other file and put it in it's place. Please help.