Finder columns jump back to last column

When traversing through the hierarchy and I reach some column deep enough that the horizontal scroll is active in the window. Then when I want to go back to one of the first columns by scrolling horizontal, the window scrolls/jumps back to show the last column.

I need to deselect the last objects (folders or files) to prevent the jumps.

It's like the window wants to keep showing the selection.

It happens on 2 machines, since Monterey.