I get pop up messages that I cannot dismiss, when apps trying to access Desktop or Download folder for the first time. Any thoughts?

I have a problem with my Mac Mini 2020, M1 with macOS Monterey 12.4

The problem is that every time an app wants to access the download folder or the desktop for the first time I get this pop up message.

I know this is normal but what not normal is, is the fact that either of the two buttons can be pressed. It looks like they have been pressed by changing the color of the button and then turning back to normal but nothing happens. If I close the app the pop up window stays and there is no way I can delete it. Not only that, but it stays at frond of all other windows. Even when I change to the secondary desktop (with Ctrl + right Arrow) it appears on the second screen. The only “solution“ I have found is to restart my Mac. After that I’m able to use the program and even though the button wasn’t clicked, I’m able to save or load from the Downloads/Desktop folder. 

I also tried to go to settings, security, go to the last folder (I don’t know how it is called in English because my system is in German) go to files and folders and give the app the permission to access the folders from there. But it doesn’t work. Only when I restart I can see that the folders are ticked and I can use them.

Does anybody know a solution to this like, I don’t know, delete a cache or something and fix this problem?

I also get the same problem when an app closed unexpectedly.