Are there standalone products that act like target mode?

Neither of those Mac's support Target Display Mode.

What you need, is an aftermarket external display for the M4 Mac mini.

see > Connect a display to Mac mini - Apple Support

+1 on Dialabrain's post. You need to purchase an actual external display, period! There is ZERO reason not to, external displays begin at <$100 and a very high quality 27" display can be easily found on Amazon for about $300.

There's no good way to use that 2017 iMac as your one and only display. Time to get a real monitor.

27" 5K monitors are on the expensive side – with street prices ranging from maybe $800 USD to $1600+ USD.

27" 4K and 2.5" monitors are a lot cheaper, and you can get some 27" 4K monitors with IPS panels and near-100% or 100% coverage of sRGB for as little as as $300 - $350 USD.

You cannot use your 2017 iMac as a monitor, period. You need an actual monitor.

Well ... you could enable Screen Sharing on the M4 and access it from your iMac, effectively using your iMac as the display for your M4 mini. The M4 mini can run headless. No extra software or dongles are needed. They just need to be on the same network and use the same Apple ID.

Basically, you enable Screen Sharing on the M4, then from your iMac use the Finder to navigate to the M4 on your network and click Share Screen instead of Connect. This runs best via ethernet because wired ethernet is faster than WiFi.

The M4's shared screen would show up as a resizable window on your iMac and you can use the iMac's keyboard & mouse to control the M4 mini. (Note, I do not advise using Screen Sharing when you update or upgrade macOS on the M4 mini, as that may not work properly or at all.)

What version of macOS are you running on your 2017 iMac? The 2017's mostly came with Sierra but could be upgraded all the way to Ventura.

The simplest solution is to find an old VGA or DVI HD display and connect this using the appropriate USB adapter (cheap). I have a second display connected to my M1 iMac using this method and my old DVI HD display is fine. You can of course use an HDMI display with higher resolution with the appropriate USB adapter.