Tcake, your solution does NOT work for everybody! Only because you say the same thing five times does not make it more useful.
In my case, the spinner spins in the Settings -> iCloud section but my contacts would NOT update as a result! I also cannot refresh my contacts like others seem to be able in the Contacts -> Groups section by pulling the screen down. In my case, nothing happens then, no spinning spinner, no freezing spinner, not anything, nada, nix... Which is all very frustrating!
After reading online almost an hour and playing around with my phone, iCloud.com account and contacts on my Mac, I finally managed to get my iPhone 6s contacts refreshed only by logging out of my iCloud account on the iPhone, selecting "Delete all data" or whatever it said at the time, and then signing back into my iCloud account afterwards. That was the ONLY thing that worked for me...
Sadly, I still cannot manually refresh my Contacts in the Groups section by pulling down the screen, so I fear that my solution might only have worked now and that it might not mean that my contacts will sync automatically again like they used to before the upgrade to El Capitan and iOS 9.1.
Come on, Apple, you must be able to figure this out - this is a huge nuisance to all of your users = paying customers!!