Siri Remote (2nd generation) clickpad gestures stop working

After normal use, swipe gestures on the clickpad can stop working. This occurs roughly once a week and the Apple TV needs to be rebooted to restore clickpad gesture functionality on the remote.

This might be related to the Siri Remote firmware update in August, it didn't seem to be a regular problem before then.

Siri Remote firmware 0x0061

tvOS 15.1 (19J572)