Exchange Calendar Sync makes watch unusable

Ever since first installation the watch has been virtually unusable. UI was pretty much jerky most of the time. Calendar and other apps sometimes take forever to startup and load content. Worst part, iPhone gets really hot and drains battery, same on the watch side. The watch battery would not last for more than 4 hours without lot's of interaction. It all seems like there is something going on in the background that makes the processor on both sides constantly run at high performance. My simple test is running the Stopwatch. It runs smoothley just after a reboot, after a while it would start moving very jerky.

I have done all kind of test disabling and enabling various features. Turns out what makes the problem go away and the watch works flawlessley if I disable the calendar within my company exchange account. Every other calendar is fine, All other exchange features like e-mail or reminder are fine.

Does anyone have an idea what could be happening with the exchange calendar that causes the watch and iPhone processor to go crazy?