iCloud Photos Not Downloading to iPhone

I just upgraded from a 6s to a XR and restored from an iCloud backup that I had done earlier that day. I have 50gb of iCloud storage and I’m only use of 10gb of it. I have 128gb on my new phone and am only useing 28gb of it. So I have more than enough room in both iCloud and on my phone for pictures so that should be of no issue here. I have my photos set to download and keep originals. But for some reason none of my photos are actually downloaded on my phone as I would like them to be, the full resolution versions are in iCloud still. If I try to turn off iCloud photos it tells me that 600 something photos were not able to be downloaded to my phone and the full size versions will be erased. How can I get them to download to my phone.