Dual monitors on my Mac Mini keep switching

It is computer sleep that is messing with the displays.

Go to the Energy Saver preference, check “Prevent computer from auto sleeping” and uncheck the rest of the boxes and test. Then with those settings, you should be able to allow the displays to shut off at a lesser time and then turn back on correctly.

What year model Mac Mini, what displays and how are they connected?

Maybe try moving the displays (right to left/left to right) or swapping the connections if that’s possible.

Could you post a snapshot of your Energy Saver Preferences please?