I think I may have found the issue.
I went back to my 2017 MacBook Pro 15" from the 2019 MacBook Pro 16" through a time machine restoration. Then the 2017 MacBook Pro started to do the same thing. That became very confusing as the 2017 MacBook Pro did not do this before I upgraded.
I worked out that my 2017 MacBook Pro did not use the dynamic wallpapers, but the 2019 did (as I left everything on default during the upgrade). But when I went back to the 2017 unit from the 2019 unit, the dynamic wallpaper came across via the time machine restoration.
So I went back to the 2019 unit, turned off the dynamic wallpaper, changed it to something static and the fan speed dropped. Restart the system as well. If you don't restart it can take a while for the fan to drop, i'm thinking there is some sort of memory leaking here which the restart flushes out.
I'm thinking that the dynamic wallpaper + external monitor has some sort of software bug. Note that i'm using a Dell monitor via USB-C to DisplayPort, maybe the Apple monitors don't have the same issue.
It would also explain another weird issue, when connected to an external monitor and in sleep mode, the 2019 unit kept running hot with high fan noise. I resolved it by disabling power nap in the power settings.
What i'm guessing is that when connected to an external monitor, the computer keeps using power nap to update the dynamic wallpaper in sleep mode. So dynamic wallpapers seems to cause fan noise when awake connected to an external monitor as well as in sleep mode.
The behaviour seems to happen to the 2017 unit as well when I switched to a dynamic wallpaper.
Can the people who are having this issue give this a try and see what you get? Also, maybe what screens and connectors people are using?