Using smart tempo without making a mess

I have an original track, to which I have added overdubs with slightly iffy timing. The original recording has no tempo information. I want to treat the original track as "set in stone", derive a tempo map from it, and then conform the overdubs to the tempo. I have set tempo mode to "adapt" and I'm looking at Smart tempo for the original track alone. Just to be sure, I select "Remove original tempo and analyze again", and it makes a tempo map that's not too awful. Unfortunately the overdub tracks are shifted all over the place relative to the original, and even tracks recorded simultaneously go horribly out of time. The result is completely unusable.

I know that smart tempo has the ability to use multiple tracks when making a tempo map, but I don't want timing issues with the overdubs to affect the tempo, so I don't want their tempo to be taken into account when creating the map. At the same time, their audio should maintain its exact timing with respect to the original, as it was recorded. My aim is to use flex time to tweak the overdubs to be perfectly in time with the tempo map created from the original track.

What's the right workflow/approach for this?