I had a same problem. After long search I have found this from user matt engops on a different forum and this will definitely fix this issue for you. I'm pasting the answer that helped me.
"What you are seeing is due to how one of the fundamental protocols that make up communication on the internet works. Typical web traffic is transferred using a protocol known as TCP, which is a method of transferring data back and forth where the connection verifies that you have received everything. It would be pretty weird if you were trying to load an image or a webpage and parts of it were just missing, right? TCP sends something called "acknowledgements" or ACKs to let the each side know as parts of the data transfer complete, so that if packets arrive late or get lost along the way, TCP can put them back in order or re-send anything that was lost.
But...
If you pin your upload 100% with iCloud or Twitch uploading or whatever, you can't send these ACKs and your download will slow to a crawl as TCP will slow itself down thinking you have a connection issue. This is why if you have a program that does a lot of uploading, if it has the ability to set download/upload limits you should set your upload rate to 75-80% of your upload speed."
In other words you have to limit your upload speed on your router if your using TCP. I limited all my iCloud devices and this fixed the problem.
