Are you running the TV in extended desktop mode, where you can set its Displays Settings "resolution" without any regards to the resolution of the internal 24" 4.5K Retina screen?
If this is a UHD 4K TV, are the adapter and cable rated for UHD 4K? (A UHD 4K TV has four times as many pixels as a 1080p TV, and really old HDMI cables designed for 720p/1080p HDTVs may not be adequate for a UHD 4K set.)
Recent versions of macOS do not like to see display transmission errors, and will deliberately cut resolution or signal if they see any. If there is a long cable between the iMac and the TV, perhaps signal attenuation, or RF interference, are causing display transmission errors that might trigger the problem.