Stuck 'Waiting for changes to be applied', takes 2 hours to START syncing

Every time I sync my iPhone 4S with iTunes, it goes fine up until 'Waiting for changes to be applied'. I dread having to update content on my iPhone on my computer because I know it will take ages. Every time I sync, it gets stuck on 'Waiting for changes to be applied' for about 2 hours, then it will START adding the new content I want (which is not a significant amount). I have tried restoring from backup, I even wiped the phone and started fresh, and still, to update my music library on my phone, it takes me a few hours. I am not talking updating hundreds of songs, I am talking a few dozen songs or maybe adding a few podcast episodes or audiobook chapters.

I have a large iTunes library so I have checked off genres and artists I want synced because there would be no room for the entire library.

What causes the sync to get stuck on 'Waiting for changes to be applied' for so long. What (if any) solution is there to this issue? Would really appreciate any tips.