How to get higher resolution on 2019 27-inch iMac being used as an external monitor for 2023 16-inch Macbook Pro

Your 2019 iMac does not support Target Display Mode, and your 2023 MacBook Pro does not support using any iMac as a Target Display.

Thus I suspect you are using AirPlay to Mac. Your Macs are new enough to support it. Unfortunately, it is not the same as a hardware connection, and I believe that it doesn’t offer 5K (or Retina modes that take advantage of 5K) as one of its resolution options.

There is a third-party product called Luna Display (by Astropad) that I believe does support 5K resolution (assuming you get the USB-C version and run an additional cable between your Macs).

Like AirPlay, it is not as good as a first-class hardware connection to a dedicated 5K monitor. You would not want to use it for applications involving a lot of fast-changing video content.

But if you want 5K resolution, it might be the only game in town, short of buying an { Apple / LG / Samsung } 5K monitor.

Another option would be to buy a 27” 4K monitor. There are some basic ones that have IPS panels and near-100% coverage of sRGB, but go for only $300 to $350.