Magic Trackpad less responsive

I have had a Magic Trackpad for years, but never really used it intensively until now! I bought a Thunderbolt Display and wanted to start using my Magic Trackpad to create a desktop setup. This works most of the time.

Unfortunately the Trackpad has a slight delay in movements every other moment. Basically I don't have the issue 90% of the time, but when it does occur, it quite annoys me a lot. The main problem is that the movement I make on the Trackpad are sent through with a delay, so the pointer on the screen is behind on my movement.