After having this issue for more than 2 years or something like that.. I finally fixed this problem, but you have to be willing to accept the fact that it will delete all your tasks! By starting all over, everything is fast and smooth again and sync will be immediately again.
You can fix the issue by going to Setting > Apple ID > iCloud > Manage Storage > Reminders > Delete Data... and confirm the Delete dialog. After that all your Reminders are gone from your devices, but when you put them back in it is finally snappy again!
Because there is no back-up method for Reminders, I made a "back-up" by making screenshots of all the lists I had. This way I remember what to put back into the Reminders app.
Hope this helps someone else!