That iMac doesn't support Target Display Mode. No iMac with a Retina screen does. The feature went away when the first 27" 5K Retina iMac came out in Late 2014 – and never returned.
That iMac is too old to be an AirPlay Receiver for AirPlay to Mac.
There are workaround products such as Luna Display (Astropad) and DuetDisplay. The only one that I know about which supports 5K resolution is Luna Display. It is not recommended for applications like video editing and gaming. Even if you were just wanting to use your 27" 5K iMac as a second display for less demanding stuff, you'd probably need to tie up two ports on your new Mac for the Luna Display USB-C "dongle" and for a cabled connection to help carry video data.
Given the high prices of the 27" Apple, LG, and Samsung 5K displays, you could look into 27" 4K displays. One of those has just enough resolution to run in Retina modes, and they're a lot cheaper than the 5K models. You might be able to find basic 27" 4K displays with IPS panels and near-100% coverage of sRGB for as little as $300 – $350 (plus possibly the cost of a separate Webcam and pair of computer speakers).