Catalina Photos : “Please unlock iPhone” message even though iPhone is already unlocked

I have searched and find a lot of people seem to have this issue. I am on a iMac Pro running Catalina 10.15.4 & I have iPhone X. My phone shows in Photos but no images appear most of the time. Sometimes just empty boxes and rarely after a long wait images will appear. I don't know if I got lucky but this last time I opened Photo on my phone and the images did appear as expected. I tried this a few times and so far it works.