My contacts keep disappearing

So, suddenly over night about 3/4 of my contacts had disappeared. Luckily I made a safety copy with iTunes on my Mac 2 days ago, so I restored my phone (loosing all texts and changes made the last 2 days in the prosess). My contacts were back, only to disappear again after a few minutes. I have done this 3 times now, and it's driving me crazy! First they all disappear, then about 1/4 of them return.

The only thing that I have done in the past day that I have not done before is buy a book on iBooks, but I don's see how that would mess up my iPhone 4's contacts... No new updates or change of settings, they just disappeared for no reason.