Upgrading Radeon graphics on 2017 iMac

Please do "About this Mac..." from your Apple menu and tell us what graphics hardware you have. We may be able to make suggestions for improving performance without your buying a new logic board. Example form my 2017 5K:

NOTE: For security reasons, please DO NOT post your seril number.

However, before doing anything else, make certain you have not created software interferences. If you have been duped into installing useless fearware like anti-virus or so-called "cleaning" apps, those negatively affect performance system-wide and could slow game play.

A 2017 iMac 5k supports certain external graphics processor units (eGPU):

Use an external graphics processor with your Mac - Apple Support

One of the problems with eGPUs is there is a 20-40% performance penalty when rendering to the internal display compared to using the same GPU in a 2019 Mac Pro. If your baseline is the top end Pro 580 then that is like having a Vega 56 eGPU. You would have to jump all the way up to to something like an RX 6800 eGPU to significantly improve your frame rates. For new hardware that is going to cost over $700!

If you can get the parts used at a good price feel free to go for it, but I don't think the expense is worth it for a quad-core i5 that doesn't have Apple support for new MacOS updates going forward. You may need to lower your gaming expectations and select lower settings and/or a lower resolution.

P.S. Do you have a fusion drive or an SSD? If you have the fusion drive it is plausible the issue may be loading assets/resources/data from disk which could be fixed by running the game off of an external SSD instead.

It is only indirectly possible by purchasing and connecting an external GPU (eGPU). However since they only work with Intel Macs and not Apple Silicon you would not be able to reuse one if/when you upgrade to a new Mac. 2024 may be too late to consider buying one for a 2017 iMac.

Which iMac do you have? (GPU? CPU? SSD or Fusion drive?)

No, not possible! iMacs are not like tower pcs that use individual cards. The GPU on your iMac is a chip soldered to the main logic board.

It's not worth sinking any money into a 7 year old machine.