What happens to my photos when buying a new Mac Mini and formatting the hard drive on my old iMac?

Three things -

1. "Optimise Mac Storage" or "optimise iPhone Storage" are fully automatic and there is no way to control when or what what get optimised. It depends on the available storage on the devices. Photos will try to keep the most recently used or imported photos stored locally, as space permits. But if you start over with a new, empty Photos Library after signing out of iCloud Photos and creating a new library and making this your system photos library and enable iCloud Photos with "Optimise", the library will initially fully optimised and as small as possible. But Photos will try to keep any photo you are using and downloading from iCloud at the full resolution, as long as you are not low on storage. Or photos you import to a device will not be optimised on the device you imported them wo, unless Photos needs to free storage.
2. After starting over with a new, empty local library and downloading it from iCloud will restore all albums and folders and metadata, but your projects will be missing. You need to keep a copy of your library (not optimised) to save your project, in addition to the regular Time Machine backup. The Time Machine backup does not include the original photos of the optimised versions.
3. Deleting a photos library from a device using the Finder or reformatting the drive does not delete any photos in iCloud. It just removes the library package completely and nothing will be synced to iCloud or other devices. The deletion of photos and videos from a Photos Library will only sync to iCloud and other devices, if you are using the Photos.app to delete items from the library.

Firstly make sure you have a backup regime for your iMac, ideally Time Machine.

Second, using Time Machine you will set up your new Mac. This will load your old Mac's system and contents to your new Mac. Part of setting up the new computer is signing in to iCloud at which point it will download your iCloud Photos, contacts, calendars and so on.

After you are satisfied that all your content, including photos, is loaded on your new Mac you will sign out of your Apple ID services on the old iMac. This disconnects your old iMac from iCloud and that computer cannot interact with iCloud Photos. Then you can erase the drive of the old iMac.

On my old iMac the library is 27 GB - is that still just the "device sized versions" and not the full versions? Seemed a lot of GB:s to me.

This sounds line you have full-sized photos on the iMac. Go into Photos > Preferences> iCloud and choose either "Download originals to the Mac", or "Optimise Mac Storage". The first leaves full sized photos, the second will give you optimised - ie. smaller - sized photos.

*This setting is in Big Sur which you will not have on your 2011 iMac. 2014 is the oldest iMac capable of loading Big Sur. There should be a similar setting in your earlier MacOS version however.