It sounds like you are trying to use Target Display Mode. That won't work.
Your iMac – the first 27" Retina 5K iMac – does not support Target Display Mode. Target Display Mode went away when that iMac came out, and no iMac (Retina or otherwise) ever had it again.
Apple also placed restrictions on the video source. Your 2017 MacBook Pro is old enough to support using a iMac that supports Target Display Mode as a TDM. But not when running Big Sur. You'd have to find an older iMac (one without a nice 27" 5K Retina screen) to use as the Target Display – and downgrade your MBP to Catalina or lower.
Read the bad news here: Use your iMac as a display with target display mode - Apple Support
All 2017 MacBook Pros can be upgraded as far as Ventura – but no further. You may want to upgrade to Ventura. There isn't much advantage to sticking with Big Sur – it won't run 32-bit applications, and it also isn't one of the "most recent three" supported by vendors like Microsoft and Adobe. After Sequoia comes out later this year, the most recent three will be (Ventura, Sonoma, Sequoia). So if you upgrade your 2017 MBP to Ventura now, you will have a year or so before Ventura, too, falls off the "most recent three" list. But upgrading to Ventura rather than downgrading to Catalina or easier means no support for using any iMac as a Target Display.
To upgrade to Ventura, back up your 2017 MBP, then see: How to download and install macOS - Apple Support .
I hope you didn't run out and buy a $50 Apple Thunderbolt 3-to-2 adapter and a $30+ Thunderbolt 1/2 cable (the hardware you would have required to make a proper Thunderbolt connection). You could use that adapter / cable for (1) creating an IP networking link between the Macs ("Thunderbolt Bridge"), or (2) using one of the Macs as an external disk drive for the other. But not, in this case, for Target Display Mode.