I think restoring is not a way to get back your contact in iCloud * i think its not possible physically .
There are some other possible ways to do this ...
You may restore bit ago and did this in hurry and yet your contacts was not loaded * it will be better to be connected with Wifi even after restoring process * this will make sure that your all iCloude data loads up properly .
Other possible reason that you had more than one accounts that had Contacts switched on.you will have to Make sure that no your other account has your Contacts toggled on. its very essential.you will have to Make sure that simply iCloud has Contacts toggle on. Settings> iCloud.this thing happen when user's other account tries to restore the contacts at same time.
If you dont have time constraint * or what else you do..it will not be possible to get back your contacts *simply delete your iCloud account and then add this again. Follow Settings> iCloud> tap the red "Delete Account" button. Restart your device.
Add your iCloud account again.
Make sure contacts is toggled on..
And certainly you make sure there should be no other accounts have Contacts toggled on before deleting and re-adding the iCloud account.
For more info visit: http://bit.do/recover-iphone-contacts