External display with USB-C to HDMI adapter has no signal after every restart

I have a secondary display connected with a USB-C to HDMI adapter. Every time I restart my iMac (Retina 5K, 27-inch, 2020), the secondary display reports "No Signal" until I physically unplug and reconnect the adapter. I must do this every time I restart, and sometimes after the computer sleeps. I am worried I'm going to wear out the port with frequent plugging/unplugging. It needs to last me for the next 5 years or so!

I used the very same external display with my previous iMac (2014 model), using a Thunderbolt 2 to DVI adapter, which worked without any issues. The display has both DVI and HDMI ports, and since I had to get a new adapter anyway (new iMacs have USB-C instead of Thunderbolt 2), I assumed that going with HDMI would be better since it is newer. Well, apparently not.

Is there any way to prevent this from happening? Or to make the external display work without physically unplugging? I tried the "Detect Displays" trick in System Preferences, but it doesn't seem to help.