MacMini, Ventura - wallpapers reset constantly. Or is it a Desktops issue?

I know this is an issue in many forms out there, I've read the posts. Most of them are however either without answers or regard different equipement which may or may not be important.

My issue is this.

I work a lot with many Desktops:

I have 4 ones set up and additional windows show up when I put an application into fullscreen mode. During working they get moved, desktop 3 becomes desktop 2 etc. What is happening is that for some reason my "constant" desktops get mixed with the temporary (fullscreen) ones, which receive a default Ventura wallpaper. On the screenshot you can see that Desktop 2 and 5 are the ones that are left after some time of working.

So the issue is that wallpapers/Desktops get screwed up. In my mind the way it should work is that, I have my constant desktop defined and they simply change places but never loose their assigned wallpapers. It is not the case however. At some point (can't say when) constant ones get replaced by temporary once with the default wallpaper. To me, it's a bug.

It has to be a desktop thing. Anyone has any idea what is the problem here and how to solve it?