That iMac does not support using Target Display Mode. No Retina Mac does. Apple removed the feature when the first 27" 5K Retina iMac came out in Late 2014. Even if it did, Apple added restrictions on "the other Mac" – which would keep a Mac Studio from using it.
That iMac is also too old to act as an AirPlay Receiver for AirPlay to Mac. In the case of an iMac, that feature seems to be reserved for iMacs introduced in 2019 or later, running macOS Monterey or later. Note that AirPlay to Mac is a software-type solution. I'm not sure if it takes full advantage of 5K displays even on 27" 2019 and 2020 iMacs.
So you're stuck in the middle – neither here nor there. There are third-party products (Astropad – Luna Display; or Duet Display) that you could explore. If those worked, they would let you use your 27" iMac as an extra display for the Mac Studio – with possible drawbacks related to trying to drive a 5K display via software. But the price of 27" 4K monitors is making those workarounds less and less attractive. Especially since you would still need to have at least one real hardware monitor.
References:
Use your iMac as a display with target display mode - Apple Support
System requirements for Continuity features on Apple devices - Apple Support
Set up your Mac to be an AirPlay Receiver - Apple Support