UI getting tiny when I use 4k resolution on Mac mini

Apple expected you to use this SCALED pane. It sets the inherent resolution of the display to its maximum natural resolution, the scales only TEXT. text is drawn much larger into an off-screen buffer, then scaled down (preserving clean edges and smooth diagonals) to be added to the active screen buffer:

You do not really say what size your monitor is. For example is is a 24" monitor, a 27" monitor, 32" monitor, 40" monitor, etc...

When you have a large number of horizontal pixels in a smaller monitor, everything looks smaller.

I got a 32" display, which I run at 3360x1890, because when I run it at the maximum resolution (3840x2160), the menu bar elements are too small. And for various apps, I've tweaked the minimum font size so that the text is readable, or I go around hitting Command+ to increase the font size in the display window.

And my 14" laptop display is set to 1512x982, even though that display can go up to 3024x1964. Cramming that many bits into a 14" display takes the eye sight of youth, and I am long past youth 😁

If you run a UHD 4K display in Retina “like 2560x1440” mode, the Mac sizes text and objects as if the display had 2560x1440 pixels. It draws in detail on a 5120x2880 pixel canvas and downsamples the contents of the canvas to paint the actual screen.

This is normal/to be expected. I also use the 2560 x 1440 resolution (on my 5K screen). The resolution marked as default (i.e. 2560 x 1440 (Default)) is the one recommended by default by macOS.