There's a few reasons why that can happen... Is the paper with two dots in the same group as the tracking shape? You can also try turning on Fixed Resolution for the group with the tracking shape (select the parent group, and in the Group tab in the inspector, you'll find Fixed Res with a checkbox.) Is the project 3D? Using a camera? (Try setting the individual groups to 2D.) Have you tried manually aligning the track over the dots (use the Adjust Item tool [I think!] and select the track and drag it into position aligned with the dots -- go to the first frame of the tracking for that.)
As animation goes on, a group can dynamically change size and position and if these things don't "align" between groups, it can cause things to derail, so to speak. The video of the moving paper is not going to change its group size, but a shape moving around the canvas (and rotating, etc) can change the group "dynamics" — especially if any part of it goes off the canvas.
The reason you are seeing the drift is the way you have setup your scene. It looks like you applied the match move to your line shape, then tracked the left point as the anchor and the right point as the rotation ,or vice-versa. The problem is the rotational center for your background does not match the rotational center of your line shape. To see if I'm right, move the line, so the center (anchor point) is on top of the point you set for the anchor in the match move and play that back.
What I would expect you would see is that the line, while offset, will now match.
To fix this you can simply change the anchor point for the line so it is at the end and then reposition the layer so the anchor for the line matches the anchor for the tracker.
It's a bit tricky to understand, but for roataional tracks like this to work (in any tracker) you want the center of rotation in the item to match the center of rotation for shot being tracked. To shoot something like this in the future, you would want 3 points (one point in between two others). You would make that the Anchor" an then select one of the outside points for rotation and scale.
Another solution with a shot like this is to use the track points behavior and "pin" each control point in the line to your tracking markers.