I'm guessing that you are using iCloud Photos to synchronize your Photos Libraries on your Mac and iPhone-- is that right? Are your devices set to "Optimize Storage" or to "Download Originals?"
My transfer of pictures from my phone to my Mac can sometimes be quite fast, and sometimes it takes half a day. Sometimes, if I'm in a hurry for some specific purpose, I airdrop pictures to my Mac and use them from the Downloads folder. They become redundant when Photos later makes the transfer.
Have you looked at how quickly the pictures show up at iCloud.com? You may see that the connection is faster to iCloud rather than from iCloud (or vice-versa.)
Since I'm often busy with Photos, the app delays transfers to keep from slowing me down. Obviously there needs to be a strong internet connection-- sometimes the WiFi that I use away from home appears strong, but the internet connection is problematic. And the OSs will try to avoid too much battery drain, so transfer from iCloud will be faster if the devices are plugged in.
Those are the major things that we have any control over, I think. Plug in, connect to the internet, and don't have Photos open and in front.