Spoke too soon, removing the LaunchDaemons didn't quite fix the issue.
So, let me clarify my expectation here though, when I connect a single monitor and work for a little, and then connect both monitors, I want Mac to organise the screens in the way I set up when both screens are present.
However, let's say I ran around with my Macbook, then came to the station, plug one screen in, worked a bit and then plugged the second screen, then Macbook completely lost its mind now. It has swapped screens exactly the way I don't want them to be. If I removed/changed cable order via behind screen (to simulate changing monitors, as I use one USB-C and HDMI in my Macbook pro), Macbook doesn't seem to understand. It still want to keep screens exactly the way I don't want.
Ideally Macbook should identify screens uniquely. And have "profiles" of combinations of this screens available, allow user to clear these organisations as they prefer (to not clog the O/S) and to keep things tidy and performant. When an existing combination of screens (combination of serial numbers or hardware IDs) connected, correct profile should be picked up and screen organisation from that profile.
Simple thing to do, but looks like that design hasn't gotten there or lost along the way. I see everyone talking about this since 2016! For some reason...